8.10 Desription of important GConv functions

Function convert()

Source in file: gc.c

int convert(size_t impnum, struct partition *imported, size_t expnum, struct partition *exported, size_t spcnum, struct space *free_space, int badblock_check, PROGRESS_PARAM, ERR_PARAM)

Converts partitions from beginning state (imported partitions) to destination state (exported partitions).

The convert() is divided into a few other functions, some less important of them are also in the same file and are named gc_* functions or GC_* macros.

• Initialization. Write various debug messages. Cut free space blocks to have at most double of blocks in imported and exported partitions.
• Check types by gc_typechecks()

  Logs some constants and sizes of some important structures, checks if they aren't too large, they are limited by the size of block (BASIC_BLOCK)

• Initialization of the first level GConv services

  There are some GConv services that may be now initizalized. These are for uniq device handles, device intervals, swapping program generator (translation tables), for details see gc_services_1(). Provided by macro GCS_INIT(1).

  Note: There are many such initializations and these are bound by dependencies, therefore there are defined macros GSC_INIT() and GSC_DONE() and argument is level (number of group). These init and done calls are described in gc_services_*() functions in pairs and are written in init-order, done-order is backward.

• Initialization GConv internals in each partition

  Mallocated item pt_gc in struct partition in imported and exported partitions and filled its items, e.g. unique filehandle, flag if is inited, location on virtual device etc.

• Check on bad blocks (optional)

  The level of checking is badblock_check, checks for badblocks (read only, read/write test or disabled) each imported and exported partition, adds badblocks into a list.

• Making the virtual device

  All areas (imported and exported partitions and free space), uses device interval functions (devint.h), adds these in list and function devint_checks() checks if given partitions are correct (e.g. exported partitions can't intersect), sorts these and sets up location on virtual device in each partition and free space.

• Changes in superblocks

  This is accelerating feature. If may be changed superblock only (there isn't any influence to structure of filesystem) then is called filesystem-dependent function fc_superblock(). If it is unsuccessful then it continues in common way. The last action is syncing devices.

• Initialization imported filesystems

  Function gc_initfs(), sets up using cache and calls fc_init() for each imported partition and sets up attribute gd_init (means partition is initialized). (attribute pt_cache_on can be changed in fc_init())

• Initialization of the second level GConv services

  GConv initialization system is described together above (first level of GConv services). Initialized various caches - cache for converting current (real) and destination (new) addresses on the virtual device (n2r), cache of the tree (tc), minicache for caching status of blocks (for very fast access, see map_status.c) (mc), support of creating the tree (trt), giving free blocks (free) and cache for exported partitions (ec). For details see gc_services_2() function, this is called in macro GCS_INIT(2).

• Create the tree (the third level)

  This is initialization of the third level of GConv services. Currently here is creating the tree only. For this the gc_services_3() calls function tree_create().

• Mark read-only (imported) partitions

  Imported partition have attribute pt_ronly, if attribute is set then the partition can't be modified in anyway (includes unused blocks). Implemented in devint_ronlyflags().

• Mark all blocks of imported filesystems on the virtual device

  Marks all whole imported partitions, follows marking all remaining free blocks (as free blocks).

• Copy filesystem specific data

  Filesystem specific data (e.g. bootsector) can be passed through this function. struct partition contains item pt_save_fsdata, item is initizalized in fc_init(). This is passed to exported partition if imported and exported partition have the same filesystem type, otherwise this is NULL. See function gc_copy_fsdata().

• Create exported filesystems

  Called fc_mkfs() for each exported partition, except partitions converted by call fc_superblock() above, sets up attribute that partition is initilized. Follows putting found badblocks to lists of exported partitions.

• Align files

  All blocks of files in imported partitions, which can be passed to exported partition as are, are marked (ALIGN flag). This minimizes number of moves (of blocks). Provided by function align_files() (found in movefiles.c).

• Move files

  All files from imported filesystem are moved into destination partitions. In current version files from i-th imported partition are moved to i-th exported partition. Moving is virtual only and files stay their locations. The metadata of exported filesystems are built only. Provided by function move_files() (found in movefiles.c).

• Close all partitions

  Close all imported and exported partitions, used call fc_done() for imported and fc_fsdone() for exported partitions, freed all GConv internal data in struct partition. See function gc_donefs()

  Swapping program must be stored on device in free blocks. Function make_transchain() computes number and reserves them.

• Generate swapping program

  At first the function make_transchain() reserves some blocks and follows function make_translating_tables() which writes swapping program in run-length encoding) into these reserved reserved blocks. The swapping program is a sequence of swap operations which permutate blocks to have all exported partition on their real destination location (the goal of the general conversion).

• Done all the third and second level GConv services

  Calls done functions for all these services. These functions free used memory etc. If any partition isn't closed than close it (this can happen if an error occured).

• Final permutation of blocks

  If all actions were successful then would be called final permutation algorithm. This algorithm is implemented in crazy_algorithm() This is critical section and can't be interrupted. Illegal interruption can cause lose your data.

• Cleaning

  Closes the first level GConv services and sync devices.

Function move_files()

Source in file: movefiles.c

int move_files(size_t n, struct partition *imported, size_t m, struct partition *exported, PROGRESS_PARAM, ERR_PARAM)

Moves all files from imported filesystems to exported filesystems, in current version all files from i-th imported filesystem are moved to i-th exported filesystem. We plan selecting destination exported partition for files in future versions because joining and splitting operations require this.

Note: function align_files() is simpler than this, browses directory structure of each imported filesystem and for each file browses all blocks. For each extent calls fc_align() (in destination exported partition), function returns subextent (or empty extent) which can stay its location.

Support for hardlinks and their duplicating is initialized (hardlink_init() and dupl_init()). Checks if passed parameters are correct. Sums blocks of all imported partitions (for computing progress). And follows main loop, actions are realized for all partitions which must be rebuilt (not solved by fc_superblock()).

The main loop consists of following, used call-sequece of filesystem calls corresponds with description in filesystem API documentation.

• Initialize imported and exported partition

  used fc_open_rootdir() in both cases, fills given structures to their root directories, prepares filesystems for browsing/creating filesystem directory structures

• Browsing and creating filesystem

  this loop browse through filesystem in imported partition, stack is represented as linklist, current directory is head of linklist, the loop is repeated while this stack isn't empty

  • Take next entry in current directory

    function fc_nextentry() returns next entry (behind given), this function is just one which gives entry, therefore returns first entry for just opened directory, returns zero if success

  • No more entries, do "cd .."

    all entries in current directory are browsed and created, rests to close this directory in imported and also exported filesystem by fc_close_dir, and change current directory upper "cd ..", entry corresponding to directory which was just browsed, putting entries into this directory finished, must be called fc_putdone()

  • Otherwise an entry found... copy it

    entry found, must be created in exported filesystem, differenced by type of this entry (directory, regular files, devices etc.)

    • Another entry of already copied hardlink

      if the entry is regular file and has at least two hardlinks, try to find out another correspoding entry in imported filesystem, used hardlinks_seach()

      if such entry found then find correspoding entry in exported filesystem by hardlink_gettemple(), create hardlink in exported filesytem by fc_hardlink_create() - if returned request to duplicate data of hardlink then add this into queue by dupl_add()

    • Create new entry and discern a few cases

      create new entry in exported filesystem as a copy of passed original entry from imported filesystem, used fc_createentry() and discern cases if entry is directory, regular file or other.

      • Entry is a directory

        open this directory in imported filesytem and also just created directory in exported filesystem, in both used fc_open_subdir(), add these on tops of browsing stacks

      • Entry is a regular file

        link this file with corresponding blocks - it is "virtual copying", it works with numbers (exactly extents) of blocks relatively to start of exported partition, blocks are assigned to this file (considering possibility of use original blocks), their numbers are linked with original numbers in GConv internal structures - in the Tree, required for permutating of blocks, for linking is used entrydata_iterate() which iterates calls of proc_movedata() for all blocks of entry

        if file is hardlink (first occurance) then this entry must be registred by hardlinks_addtemple() (to hardlink other links to it)

      • Other file

        e.g. device, pipe or softlink, it doesn't need putting data or any other actions, all required actions are done in fc_createentry()

• Duplicating process

  duplicate all hardlinks which are requested to be duplicated, were added by dupl_add(), used dupl_process() function

  follows freeing resources used for duplicates and hardlinks by functions dupl_done() and hardlinks_done()

Function entrydata_iterate()

Source in file: movedata.c

int entrydata_iterate(struct partition *imported, struct entry_info *imp_entry, struct partition *exported, struct entry_info *exp_entry, int (*proc)(struct block_list *list_glob, struct block_list *list_exp, struct partition *exported, struct entry_info *exp_entry, ERR_PARAM), ERR_PARAM)

Its task is to pass all (extents of) blocks of entry in imported partition to given function to do something with these blocks in exported partition, e.g. decide about destination location of this entry and link source and destination numbers of blocks for later permutating of blocks.

Function consists of double loop, gets blocks from source entry imp_entry, these blocks are relative to to start of imported partition. Blocks are renumbered to be relative to start of the virtual device (it contains everything) by extents_globalize(). Blocks are passed to given function (from argument) exp_entry which can consider exported partition and exp_entry. Function move_blocklist() converts blocks to be relative to start of exported partition or marks that are outside (used constant INVALID_BLOCK as first block). Such list can be fragmented, function join_blocklist() joins all neighbouring extents which can be joined (satisfying logical order). The most important action is call passed proc function with these lists to do something like linking, aligning etc.

The last action is to free used resources by fc_getdone() for imp_entry and exp_entry.

Note: exception for exp_entry == NULL is used in aligning proc_align().

Function proc_movedata()

Source in file: movedata.c

int proc_movedata(struct block_list *list_glob, struct block_list *list_exp, struct partition *exported, struct entry_info *exp_entry, ERR_PARAM)

Note: functions proc_align() and proc_duplicate() are very similar to this and don't need detailed description.

Consists of single loop which iterates through all passed blocks and does following actions:

• Take free blocks

  filesystem can have problems with locating new entry, but I prefer to use blocks outside files in imported partition, therefore it must give such blocks to exported partition (exported partition has infomartion only about own already created files), used offer_blocks()

• Put blocks to exported filesystem

  now I pass blocks which must be copied in exported partition, so function fc_putdata() must decide about their future location

• Patch for list of free blocks

  some free blocks were used, must be removed from list used in offer_blocks() function, function free_exclude_extent() cares about this.

• Link correspoding blocks

  original and destination numbers of blocks must be linked, this is required for later permutating, by function link_blocklists()

• Cut satisfied blocks

  cut satisfied blocks because there is possible particular success in fc_putdata() function, used extents_cutfirst()

Functions bread() and bwrite()

Source in file: brw.c

block_t bread(struct partition *part, const struct extent *blocks, void *buf, ERR_PARAM)

block_t bwrite(struct partition *part, const struct extent *blocks, const void *buf, ERR_PARAM)

Part of API, unify interface for logical access to partitions. Therefore these are usable for both imported and exported partitions. Even in special accelerated functions.

• Direct access to device

  if pt_direct_access is set then functions bread() and bwrite() access to device directly, used device_read() or device_write(), this method is intended for accelerated actions like mkfs or zeroing partitions etc.

• Access to imported partition

  attribute pt_imported is set for imported partition, if cache (for imported partitions) is enabled then is used by impcache_read() otherwise is used device_read(). Note that write to imported partition is not permitted.

• Access to exported partition

  exported partition exists virtually only (on the virtual device in its destination state), simply used function linnew_read() and linnew_write() which are intended for reading/writing to virtual device in its destination state and use exported cache, their description follows.

  Note: bread() and bwrite() function are entirely block-oriented (unit BASIC_BLOCK), but there are written byte-oriented extensions bread1() and bwrite1(), but are emulated by bread() and bwrite(), in current implementation of GConv is not better way how does it.

Functions linnew_read() and linnew_write()

Source in file: map_rw.c

int linnew_read(const struct extent *ext, void *buf, ERR_PARAM)

int linnew_write(const struct extent *ext, const void *buf, ERR_PARAM)

Provide read and write operation with the virtual device in its destination state. Functions are intended namely for accessing to exported partitions and are used in bread() and bwrite().

Functions consist of loop, which is repeated until action is done. Rest of given extent is converted from destination to current addressing of the virtual device in each iteration, used new2real(). Rests differ for read and write. Read is very simple use of exported cache expcache_read(). Note: block which wasn't already written contains zeroes. Write is very similar to read, it uses also exported cache expcache_write() and blocks are allocated during the first write by alloc_realblock() and also these must be linked with current destination, e.g. for filesystem metadata.

Function crazy_algorithm()

Source in file: crazy.c

int crazy_algorithm(PROGRESS_PARAM, ERR_PARAM)

The function does modification of all partition which are consireder for conversion. The functionality is very simple, works in single for-cycle for all blocks. In i-th iteration of the for-cycle function get_next_swapop() returns number of another block and these two blocks may be swapped or second one may be copied on first one (depends on returned moveonly flag). If such action is required (means that second one differs with INVALID_BLOCK) then function crazy_swap() is used to do it. The crazy_swap() is buffered and such buffering makes this simple idea very fast.

Note: simplicity is very important to be possible to implement journaling in feature versions, and to be sure that this critical code is without bugs.