wpkg known bugs

Submitted by Alexis Wilke on Tue, 04/30/2013 - 06:32

The following are inheritant bugs that the packager has problems with and will not be fixed or may be fixed in the distant future. Bug that you find and should get fixed have to be posted on SourceForge.net.

Compilation of the wpkg Solution (VC++ IDE)

When creating a solution file and running a "Build Solution", you get errors. This is because the IDE builds everything all at once and it is incorrect in our environment.

What you want to do is: right click on the ALL_BUILD target, then select Build. That will build everything, except the packages. To then build the packages, right click on the wpkg_package target, and do the same. However, the wpkg_package only builds packages for a Release build. Make sure that you first run the Build on the ALL_BUILD in the Release configuration.

Supported configurations: We only support the Release and Debug configurations. cmake adds other configurations that we do not use or support in any way.

MSVCR100.dll or MSVRP100.dll missing

We noticed that without the development environment the different wpkg tools would not start because the CRT DLLs from Visual Studio are not made available on a default system (I do not understand why that is done that way, but there must be a reason...)

This is true up to version 0.8.2. Since version 0.8.3 we use the /MT and /MTd options so that problem does not occur. However, in version 0.8.x we will not make use of the new target_compile_options() of cmake because it is not yet available in an official version of cmake. This means the DLLs are probably not going to work too well for you. If you want to use the DLLs, edit the CMakeLists.txt file and delete or comment out the set() commands that set different C and C++ flags with /MT and /MTd (around line 131.)

Since version 0.9.0 the entire environment compiles exactly as expected, meaning that static libraries and tools are compiled with /MT[d] and dynamic libraries and tools are compiled with /MD[d].

So what is the length limit of a filename?

If you are new to MS-Windows, you may be surprised to learn that the length limits are quite complicated to understand. Each software has its own limit and because of that, the limits defined in wpkg are likely not the limits you will encounter while using MS-Windows.

The main limit is the size of a path and filename. The MS-Windows API allows for filenames of up to 32Kb (or as Microsoft documents it: close to 32,768 characters because of this and that—such as the \\?\ introducer—it will always be a little smaller.) So wpkg does support 32Kb file names in UTF-16. In other words, any Unicode characters, although note that characters that use 2 words count as 2 characters...

Opposed to that high limit, most software make use of MAX_PATH buffers to get things such as the current working directory or to handle a filename. That limit is 260 minus 12 (the 12 comes from the 8.3 filename: 8 + 1 + 3 = 12.) So as a result many software have a limit of 248 characters! Especially, all the multi-byte functions, such as the CreateDirectoryA() function, do not support extended paths (paths that start with \\?\...) and many software are programmed using those functions.

Other software have larger limits, but most often defined in some sort of arbitrary way. For example, a software may decide to make use of a buffer of 1024 bytes (1Kb.) This gives you more room, but that's really only if the entire software consistently use buffers of that size or larger.

Software that I know do not comply to the 32Kb character limit are the Explorer. Yes! Trying to create files with very long names, over 220 characters, then try to copy the directory that contains that file from Explorer. Does that work for you? In Windows 7 that still fails.

The cmd.exe interpreter, also called MS-DOS window, does not handle very long paths either, although the limit is larger than Explorer, it is likely that you will run into problems trying to handle your files from and MS-DOS prompt. If you need to do so, even just verify that things are installed as expected, then use the Cygwin console instead. Note that affects the package scripts since these are run using the cmd.exe sub-system to run the .bat scripts. To go around this problem in your scripts, you can create a C++ tool to process what your script was expected to process. That way you can use the MS-Windows API and the setup will succeed.

Oh! You were wondering what the limit would be under Linux or other Unix systems? wpkg does not limit its file name length to the operating system, although it supports two command line options: --path-length-limit and --enforce-path-length-limit so that way you will know if the filename is too long when creating a package. However, if you install that package under a sub-directory, that sub-directory filename is not included! That being said, by default wpkg has a soft limits of about 1,000 characters (soft in the sense that it only emits a warning, not an error.) Linux has a limit which is 4Kb, so much smaller than MS-Windows and yet rarely would someone reach that limit if ever! (I think that if you recompile your Linux kernel you can change that limit, it may even be possible to change it using tune.)

wpkg Detaches Itself on Self-Upgrades

The wpkg tool can self-upgrade, which is great! However, under MS-Windows, such a feat is only possible if the executable that you are upgrading is not currently running. To palliate to that problem wpkg process is to:

  • Copy itself
  • Run the copy with the additional --running-copy command line option

The copy works exactly as expected. However, the running the copy version makes use of the exec() function call and that overwrites the current process with the copy. Under Unix that works as expected: the new process receives the tty of the old process, in other words, the tty is still blocked.

Unfortunately, under MS-Windows that means the old wpkg.exe tool exits before it gets replaced with the upgrade, so you get the prompt back, but the process is NOT done. There are no safe way, at this point, to know that the process is finished although there is a procedure that you can follow to avoid continuing before the upgrade is done:

  • Create a batch script
  • Start the wpkg upgrade with --install and the name of the new wpkg package
  • Wait for the first wpkg process to return
  • Wait for the database lock file to appear
  • Wait for the database lock file to disappear
  • Check that the upgrade worked by running wpkg and comparing the version with the new wpkg package version, if it matches, it worked

Note that this mechanism is not perfect because you could be blocked for a long period of time before you can detect that the database lock file exists. In that case you would miss that signal and then you cannot really assume that the tool quit again when it disappears.

We will be looking into fixing this whole problem in a later version of wpkg so you won't have to do such work to ensure upgrades. Note that if you upgrade a very large number of packages, including wpkg, then you will run in this problem too, however, you are much less likely to miss the step "wait for the database lock file to appear".

The computation of the tree seems to fail... (Before 0.9.4)

In version 0.9.3 and prior, the computation of the tree would make use of a quick and dirty algorithm that would in the end miss many possible resolutions and in some situations would prevent the system from finding the right combination. In such a case, wpkg would print an error and not install anything.

Since version 0.9.4 we calculate all the possibilities. This means we may end up spending a lot more time searching for the possible cases, but at least we'll find the right one and allow installation if there is a valid case. We incorporated a new test to verify that problem. The old version (0.9.3 and prior) would find 7 different graphs. The new version (0.9.4+) finds 72 graphs for the exact same case. This proves that the old version was skipping on many valid cases...

If you have a problem installing many packages with many dependencies automatically installed from a repository, know that you are probably hitting this bug. You have two possibilities:

  1. If you can, upgrade to 0.9.4 or better version of wpkg,
  2. Install each dependency by hand instead of implicitly (then the tree you offer is already perfect and used pretty much as is.)

Found by Trent, see https://sourceforge.net/p/unigw/discussion/647573/thread/8391a6a4/

Remote Upgrade through HTTP (Before 0.9.3)

The software runs a check on the URI to see whether it is an existing local file before using the URI. A fix was applied to skip the test for schemes other than file:.

Found by Trent, see https://sourceforge.net/p/unigw/discussion/647573/thread/17a08da5/

Description Field Checked too soon (Before 0.9.0)

The description field is checked for unwanted characters, namely the horizontal and vertical tab characters. In older versions this test happens before the field value gets transformed so if variables are defined in the field you may end up missing those tabs as intended by the test.

Infinite Loop When Replacing Variables (Before 0.9.0)

A field that somehow references another which itself references that first field creates a cyclical dependency which is not caught in versions of wpkg before 0.9.0. Say for example that you write the following:

X-My-Field-A: $(get_field("X-My-Field-B"))
X-My-Field-B: $(get_field("X-My-Field-A"))

wpkg will crash once the stack memory is full. At this point I do not plan to fix this problem in 0.8.x.

Drive Letter Support

Microsoft Windows started with the existing systems at the time: MS-DOS. This operating system viewed each accessible drive as a device which was assigned a letter starting with the letter A. Once hard drives became common, those got assigned drive C and over.

Onto one hard drive it is also possible to create multiple partitions. Each partition is assign a letter in incrementing order.

Today, Microsoft also added remote partitions via its NetBIOS protocol. So a remote hard drive directory can be assigned a letter as well.

In our case, we support such drive letters, of course, but to simplify and prevent potential problems when creating packages cross platform, we implemented a scheme where such drive letters can actually be used under Unices. The idea is simple: one code base that does the same thing whatever the platform. This is why even under Unix you can write something like C:/home/me and the path will be recognized.

As a side effect, obviously, any time you attempt to specify a filename that starts with a latin letter (a-z or A-Z) followed by a colon, it is viewed as a drive letter specification, whether or not a file of that very name exists on your hard drive. So "b:ah.txt" is viewed as "B" drive, file "ah.txt"1.

We are not looking forward to having a way around this potential problem. We simple do not expect people to make use of such filenames. Note that a version in a Debian package such as:

wpkg_3:0.8.0-4_win64-amd64.deb

works exactly as expected under Unices (i.e. the colon is preceeded by a digit and it is not the second character in the filename.)

Limitation to Filename Length (Before 0.9.0)

The old tar format was used in older version making it impossible to have filenames that are any longer than 100 characters (it could be longer with a separate path though.) This is due to the tar file format which has a name length hard coded in its header.

It is possible to get longer filenames adding a file with this special filename: "././@LongLink". This file data is the real filename. Then the next block is the actual file data with the truncated filename in the usual place.

For details information, you may look at the source code of libtar which is what GNU tar uses.

Invalid Characters ( / \ : * ? " < > | )

The FAT and NTFS file systems prevent a certain number of characters from being used in filenames. Because we want all packages to be compatible accross platforms, we also prevent those characters from appearing in filenames under Unices.

Note that out of the invalid character the slash (/) and backslash (\) characters are both viewed a file name separator in a full path. Therefore, in effect they are forbidden in a filename (Unix also makes use of the slash to separate components of a path.)

The colon (:) character is accepted under Unices because it may appear in versions. Note, however, that a colon right after a letter (a-z or A-Z) at the beginning of a filename is viewed as the drive separator, even under Unix. It is adviced that you do not use the colon except for version epochs. For such files, make sure to use a semi-colon under Microsoft windows.

The asterisk (*) and question mark (?) may appear in filenames as these are used for pattern matching (globbing and directory reads). However, a final name should never include these characters.

The other characters are simply forbidden: double quote ("), less than (<), larger than (>), and the pipe (|).

Note that these characters are not forbidden in URIs and therefore you get errors only if a direct (disk) filename includes those characters.

File Scheme Common Misuses and Bug

Older Versions (before 0.9.0)

Before version 0.9.0, the file scheme did not decode anything. The filename was taken as is which was a problem in some cases. Also the scheme did not support a domain name, not even localhost or 127.0.0.1.

Common Misuses

When using the file scheme, all the elements found in the path get decoded. This means the percent (%) character is viewed as an introducer to a two digit hexadecimal number. In order to include a % character, use "%25" to make sure it works in all cases. For example, the filename file-34%.txt should be named:

file:///file-34%25.txt

To work in all cases.

Similarly, the question mark (?) and hash (#) characters need to be escaped otherwise they will be viewed as the query strings and anchor respectively.

IMPORTANT NOTE

Note that plain filenames do not get decoded in this way. So a % remains as is and "%25" would be what the filename actually looks like.

If one of the two characters following the % character is not an hexadecimal character, then the % is used literally. So a filename such as "50%" or "%this%filename%" would both appear as is even after the decoding process.

This is nearly the normal behavior of the file scheme, but it may come as a surprise to people who think that the decoding only applies to the http scheme.

Note that we support the plus character as a replacement of the space character. This means the plus (+) character itself must be escaped with the %2B encoding.

The %2F Bug

Note that if you use %2F in a path, it will properly be converted to a slash, but the filename object will be misinterpreted. In most cases, it will work, in some circumstances it can misbehave. Just don't use %2F in the path part.

Invalid MS-Windows Names

Early on MS-DOS used special names to access certain hardware peripherals such as COM1 for the first serial port and LPT1 for the first parallel port. Under newer versions of the operating system, this is done with a \\?\... type of filename which allows for clear distinction between files and hardware devices.

However, for backward compatibility, the Microsoft operating system still supports these special names. In order to avoid any problem with packages, all those names are forbidden from anywhere in direct filenames (file://... and smd://... protocols.)

The names concerns are:

  • AUX
  • CON
  • COM1 to COM9
  • LPT1 to LPT9
  • NUL
  • PRN

MinGW Unicode Limitations (Before 0.9.0, and 0.9.5+)

Since 0.9.5, a special addition was made in order for wpkg to support Unicode in the MinGW environment. A patch was offered by Mathiew Schroeter that added a new main which makes use of the full Unicode version of the command line arguments before calling the other main functions.

Until 0.9.0 MinGW was not fully compliant with the MS-Windows Unicode (UTF-16) capabilities. For example, the ifstream does not support opening a file with an std::wstring preventing the use of Unicode (UTF-16) filenames.

If you require full Unicode support, make sure to use a version that was compiled with VC++.

Note that although package filenames are limited to ASCII (actually 0-9, a-z, underscore, dashes, and a few other characters), the path to the target installation may require Unicode (UTF-16) characters. In that case, the VC++ version is better.

We fixed the basic problem in 0.9.0 by creating our own fstream which always makes use of Unicode functions under MS-Windows. Although this is not yet tested in a full mingw32 version, it works just fine with the Win64 version compiled with VC++ so I would imagine we're good. I'll still have to add tests to make sure all parts do properly support Unicode filenames.

Upgrade Leftovers

When upgrading a package, files from the old package that do not exist in the new version of the package that are present in the database area do not get deleted. This can cause problems since these old files may still be accessed in some circumstances. Because the system is expected to make use of the index.wpkgar, it should not happen, but it is definitively tricky.

At this point, maintainers are requested to not remove old files, if not necessary anymore, render them neutral instead (i.e. a script could be made completely empty.)

  • 1. You may use the ./ introducer to avoid the drive detection. It should work in most cases, although at times that introducer is removed and a re-parsing of the filename will fail anyway.