DevdocCommit

What is happening when a downloading file finishes?

First of all some notes:
"!!xxx" means an option or other data which is stored in MLDonkeys ini files,
for example: !!temp_directory, which can be found in downloads.ini.


Well, how does it all begin?

Donkey=============================================================================================
A timer is created in donkeyMain.ml that checks every !!compute_md4_delay whether a file is complete:
      add_session_option_timer enabler compute_md4_delay
        (fun _ ->
          DonkeyOneFile.check_files_downloaded ();
          DonkeyShare.check_shared_files ()
      );

DonkeyOneFile.check_files_downloaded ()
- goes through all downloading files and calls donkeyOneFile.check_file_downloaded

donkeyOneFile.check_file_downloaded
- checks with the swarmer if all chunks have status 3 (complete and verified)
- checks with the swarmer if the downloaded data matches the file size
- calls donkeyOneFile.download_finished

donkeyOneFile.download_finished
- calls donkeyOneFile.declare_completed_file

donkeyOneFile.declare_completed_file
- among other things most important calls commonInteractive.file_completed

commonInteractive.file_completed
- works only on files with state FileDownloading
- file is moved from !!files to !!done_files (these are sections in files.ini)
- file receives state FileDownloaded
- CommonShared.new_shared is called
- file completion mail is send

CommonShared.new_shared
- puts the file into hashtable dirnames
- iters through all network modules and calls CommonNetwork.network_share

CommonNetwork.network_share
- calls op_network_share in all network modules

donkeyShare.ml
let _ =
  network.op_network_share <- (
The following code is called on op_network_share.
The donkey module has its own list of shared files besides commonShared.ml.
This is a bit confusing and should be changed in the future, shared_files_new.ini
is maintained only by donkeyShare.ml, shared_files.ini is maintained by
Gnutella/G2/Fasttrack(GGF)-module.

Here update_shared_num is called which is again in commonShared:
commonShared.update_shared_num
- adds the file in shareds_by_num
  shareds_by_num is a hashtable in commonShared which is, for example, used
  by the "upstats" command to display the list of shared files:
    driverCommands.ml
     "upstats", Arg_none (fun o ->
        let list = ref [] in
        shared_iter (fun s ->

  A list named "list" is created which is filled with the results of shared_iter
  which gives the results of shareds_by_num:
    commonShared.shared_iter f =
      H.iter f shareds_by_num

shareds_by_num is a so called "weak hashtable" which means it is only available
from the module where it is created, here it is commonShared.ml. It therefore can
not be accessed directly from driverCommands.ml, so shared_iter is used for this.

The next command in the timer in donkeyMain.ml is "DonkeyShare.check_shared_files ()" but
it works only on committed files. Committing means *moving* the physical file from the
!!temp_directory to the incoming directory. This will be explained later on


BitTorrent==========================================================================================

Another timer is created in bTMain.ml
   add_session_timer enabler 60.0 (fun timer ->
        BTClients.recover_files ();
    );

bTClients.recover_files ()
- calls check_finished

bTClients.check_finished
- checks with the swarmer if the file is finished, then calls bTClients.download_finished
    if Int64Swarmer.check_finished swarmer then download_finished file

bTClients.download_finished
- notifies the tracker that the file is finished
- calls commonInteractive.file_completed (see notes in the Donkey section)

op_network_share
- does nothing here. The finished file(s) are still in !!temp_directory waiting for commit

====================================================================================================
committing files
====================================================================================================
Committing takes place either automatically (!!auto_commit true, files are checked every minute)
or manually, both ways call commonInteractive.file_commit

Here is the manual way
driverCommands.ml
   "commit", Arg_none (fun o ->
        List.iter (fun file ->
            file_commit file
        ) !!done_files;

This code checks every minute if a file was moved to !!done_files and, if
auto_commit is set to true, commits the file, this happens in the background
driverMain.ml
  let minute_timer () =
  if !!auto_commit then
    List.iter (fun file ->
        file_commit file
    ) !!CommonComplexOptions.done_files

commonInteractive.file_commit
- here the real action of moving files takes place, an important comment from the source code:
    (********
    These two functions 'file_commit' and 'file_cancel' should be the two only
    functions in mldonkey able to destroy a file, the first one by moving it,
    the second one by deleting it.

    Note that when the network specific file_commit function is called, the
    file has already been moved to the incoming/ directory under its new
    name.
    *)
- "let incoming" seeks for the appropiate incoming directory, if the downloaded file is a
  directory (means multifile torrent) another incoming is chosen in respect of the sharing
  strategy used. The first shared directory with the corresponding strategy is chosen, so
  there is no need to have several shared directories with a incoming_* strategy.
  It does however not hurt, files to be shared can be put into those directories.
(are directories created if one of the sharing strategies is missing?)
- "let new_name" checks for the file name the commit file should have -> file_commited_name
  commonInteractive.file_commited_name
  - appends an ongoing number to the filename if the file already exists in incoming
- "set_file_disk_name" calls commonFile.set_file_disk_name
  - commonFile.set_file_disk_name
    - really moves the files via Unix32.rename
    - Unix32.rename takes care if the finished file is really a directory from a multifile
      torrent
    - Unix32.rename copies the files if renaming fails due to the fact that !!temp_directory
      and the incoming directory are on seperate physical partitions, this can block the core
      if the files are huge. It is therefore advised to have !!temp_directory and the incoming
      directory on the same physical partition
- "file_best_name"
  - take a look at files .ini
    - one option is called file_filename which is the default name for the file, normally it
      corresponds with the name present in the ED2K link or the "name" option in a torrent file
    - another option is called file_filenames, AFAIK only used by the Donkey module
      here alternative filesnames are stored which is BTW a good way of identifying fake files
(result?)
- "Unix32.destroy"
  - MLDonkey has a table of used file descriptors (fd), here the fd of the file is removed
- next line removes the empty directory of a multifile torrent in !!temp_directory
- "impl.impl_file_ops.op_file_commit impl.impl_file_val new_name"
  - this long line calls the commit function in the various network modules, keep in mind
    the file(s) is/are already moved, GGF/FileTP don´t use this function
  - donkeyInteractive.file_ops.op_file_commit
    - DonkeyShare.remember_shared_info
    - unshare_file
  - bTInteractive.ml
    - (to be understood;-) and written)
- commonShared.new_shared is called
- file_state is updated to FileShared
- file is removed from !!done_files and !!files

(what about the swarmer for the file, they are not removed from files.ini)
(what about commonSources, are the sources removed?)

=======================================================
Donkey: I noticed that that finished files are present twice in shared_files_new.ini
- once with their real name
- second with "temp/urn_ed2k_ABABABABABABA" as file name, other data like md4s are the same

This seems due to the fact that
  donkeyShare.remember_shared_info
is called twice, once via

commonInteractive.file_commit
  impl.impl_file_ops.op_file_commit impl.impl_file_val new_name
    donkeyInteractive.file_ops.op_file_commit <- (fun file new_name
      DonkeyShare.remember_shared_info file new_name

and before committing via (to be removed?)
  donkeyOneFile.declare_completed_file
    DonkeyShare.remember_shared_info file (file_disk_name file)