30 November 2011

MSYS/MinGW Porting Kit


What I use to build most binaries here? MSYS+MinGW, yet I rarely post something specific about it...

Tonight I cleaned up my toolchains and dump everything into one kind of integrated package, a porting kit I would say. Formerly it just a small set of MSYS and TDM's GCC, slowly it grew up by adding some tools for packaging/deployment. Recently I even build my own GCC (i686-w64-mingw32 multilib) based on 4.6.2 version due to TDM's GCC multilib have default 64bit target which not so parallel with the spirit of 32-bitness of this blog fu fu fu... so I compile GCC multilib that do the opposite (32-bit by default and optionally targeting 64-bit). Furthermore as with  MinGW-W64 get matured but not fully compatible with MinGW32 I made alternate GCC mimicking official MinGW32 which migrated to dwarf2 based exceptions and switch between them as needed.

Tips for complete noob (to be continued..)
- to install just run the self extracting installer, it will extract to [current folder]\MSYS, once finished run msys.bat (it is portable installation)
- to see compilers and gdb manual use "info" or "man"
- in msys "/" is the [current folder]\MSYS above and all driveletters mounted after "/" e.g "D:" = "/d", therefor "/" also equal to "/[current driveletter]/[path]/MSYS" see details by type "mount"
- to compile a source package usually we "cd"ing (change directory) into the root of source's folder then type "configure" and then "make"
- the recommended way to compile however is to create a "build folder" outside (better one) or inside source folder then from "build folder" type ../[source folder]/configure or ../configure
- to cleanup compilation files use make clean, for total cleanup use make distclean (need re-configure)
- to install use make install which by default goes to /local folder
- to change default install location use --prefix="[install folder]" to configure line
- we can pause operations in msys by drawing mouse into console (like when we select area), I often do this when my CPU rang overheated during compile and to continue right click on the console. This also equal to copying the content of selected area.
- to redirect compilation messages into file, we can use the usual ">logs.txt" way
- to force verbose compilation when it was suppressed use make V=1
- there is "cp" -> "copy", "rm" -> "del", "mv" -> "ren", or we can use alias copy=cp if that bothersome
- like most console program we may switch into "DOS" mode by typing cmd, there all environment variables will automatically converted to windows style once we're done type exit.
- path auto-completion only works in unix style e.g /path/path/file
- when unix style path feed to non msys program it will be converted to windows path style albeit with "/" instead of "\" separator
- msys is semi-case-sensitive environment where path and file are case-insensitive and most of the rest including variable name are case-sensitive
- to define global variable e.g compiler flags use export varname="value"
- in windows arguments can be represented as %1 %2 %3... similarly in msys $1 $2 $3... and for %errorlevel% in msys this is most likely $?, too bad no %~[x][varname] expansion there
- unlike windows, most command options are begin with "--" or "-" instead of "/" e.g <code>configure --help
- if we get source snapshot tarball with no configure available use "autogen.sh"
- if source packages do not use autotools (configure), try look for makefile or makefile.gcc file then use make -f [makefile] to compile. Other possibilities SConstruct (scons), CMakeLists.txt (cmake)
- to define compiler's flags while overriding global compiler's flags, feed the configure line e.g configure CFLAGS="someflags"
- to change default libtool's dll naming we may modify soname_spec variable (usually inside libtool file after configure done)
- to build an orphaned library (a library required by only a library) as static lib, use --enable-static --disable-shared in configure line
- when throwing python in compilation inside msys, things will get very ugly: path will be mixed between "\\", "\" and "/" separator and most likely fail the process. The easy fix is to actually edit bundled python's ntpath.py and change with sep="/"
- when compiling 32bit binaries I suggest use mingw32 simply because either some packages may need adjustment for mingw-w64 or mingw-w64 is not too similar to mingw32
- according to mingw project, dwarf2 is the future and it's not recommended to mix between mingw32, mingw-w64, dwarf2 and sjlj even if it's possible, so be warned
- a library usually has .a(static lib) or .dll.a(impor lib) extension, although this is just naming convention gcc will prefer .dll.a (dynamic linking) over .a file by default. More precisely: libfoo.dll.a, foo.dll.a, libfoo.a and at last foo.lib.
- if we get undefined error during linking and we find the missing function in library xyz then we can:
re-configure by append LIBS=-lxyz to configure line or
edit suspected makefile and add -lxyz to LIBS declaration or
issue manual make command with the missing -lxyz
- if we still get undefined error maybe we're linking to incorrect static library? maybe we forgot to define something like -DSTATIC_LIBXYZ? or sort of declspec stuff in the suspected lib
- to generate smallest executable (for .exe only) use CC="gcc -flto", CFLAGS=-Os and LDFLAGS="-Wl,-s " but lto (gimple) isn't matured yet and you may encountered ICE

28 comments:

  1. Wonderful! You have done some great porting. I must check out your porting kit!

    Andre

    ReplyDelete
  2. Nice build with a lot of additional libraries.

    However, compared to other builds:

    No thread support built in (for c++0x)?
    GMP, GMPXX, MPFR, MPC dynamic only?

    ReplyDelete
  3. Isn't c++0x std::threads still experimental?
    if not I will enable it in 0.3b, thanks for inform me

    The reason for GMP, GMPXX, MPFR, MPC being dynamic build is it make compiler executables (cc1.exe and friends) much smaller :D, after all it's just compiler dependencies, unless you mean that your program need them? I only supply them so the users can rebuild the compiler

    thanks for trying!

    ReplyDelete
  4. I am not sure about how experimental is std::thread, as I have used boost::thread before (and WinAPI threads before that) and after gcc-4.6.2 was released I have started using std::thread. I haven't used all the features yet but the things I used do work for me.

    The reason I was trying your build of compiler, is that the build I use (from code.google.com/p/mingw-builds/downloads/list which has std::thread support) is having problems with long double numbers (i.e. by assigning -1. the value stored in long double variable is -0.).
    It is just much easier to learn all that new features of new standard C++11 (which is of course not fully implemented yet) by having everything you need in compiler.

    I do actually use GMP, GMPXX, MPFR libraries in calculations as the precision of standard types is simply not enough, but that is not a big problem, as I can copy static libraries from other build and use them in yours as long as the version of the compiler is the same (I do hope :) ).

    Thank you for Your input into open source tools popularity.

    ReplyDelete
  5. I have re-read some mailing-list and std::thread is still not supported in mingw/cygwin due to incomplete feature in pthread-win32, personally I have tried gcc from code.google.com/p/mingw-builds but its (patched?) pthread.h cause trouble to some packages (won't even compile)...

    ReplyDelete
  6. correction: it seems not pthread-win32 but winpthreads http://mingw-users.1079350.n2.nabble.com/enable-std-thread-experimental-patch-td6733978.html

    and the reason why some package failed is pthread_t are different. Anyway it still experimental.

    About your usage of gmp and friends, please note that all of them use march=core2 flag

    ReplyDelete
  7. Looks like you are right, it is still experimental...

    march=core2 is not a problem as all my multi-threaded applications are executed on multi-core processors. :)

    Thank you for clarification of the situation.

    ReplyDelete
  8. Hello, thank you very much for providing these packages; I was looking to install myself a MSYS installation but knowing of the lots of packages that are available it's a lot of work and I stumbled upon yours.

    I would like to request something for the next release if possible. Right now you're including the libiconv binaries but not the development headers and libraries (right now is the file libiconv-1.14-1-msys-1.0.17-dev.tar.lzma in SourceForge). I'd appreciate if that could be included by default too (maybe under the "local" folder).

    I'd also like to know if you plan on updating your compiler package with GCC 4.7 which is now considered to be stable.

    Thank you very much anyway for everything and congratulations on a well done work.

    ReplyDelete
  9. There is a lot things in version 0.5 to be planned. some of them:

    - proper MSVCRxx specs mode: done (tested by building whole GTK against msvcr90.dll)
    - complete documentation toolchain: lack docbook-utils (in progress, openjade is done though)
    - LFS in mingw32 : done
    - winpthread and std::thread in mingw32: in progress
    - spkg integration : in progress
    - turn git to static build: done
    - merge clang,tcc,gcc into one folder: in progress

    anyway iconv in my gcc is a system library (so does pthread and dlfcn), hence reside in [gcc]\triplet

    Thanks for using my pacakges

    ReplyDelete
  10. Error (509)
    This account's public links are generating too much traffic and have been temporarily disabled!

    ReplyDelete
  11. knew it man, I'm slowly mirror them (not all maybe). version 0.5 will come out this weekend, o yeah with gcc 4.7.0 i hope

    ReplyDelete
  12. Git seems to be broken in the current pack, I'm trying to execute a cloning operation from Github and this is the output:

    fatal: Unable to find remote helper for 'https'

    or http in case I try it that way.

    ReplyDelete
  13. Thanks man, I though there is just another missing hardlinks for git-remote-http git-remote-https and git-remote-ftps but turn out there is also threading error too since I override git's pthread. Will rebuild that

    ReplyDelete
  14. Hello, I don't have much knowledge about the different exception models, but I read that they're not to be mixed. So I have a question, I'm rebuilding a library to be used by a program. That program uses more libraries of different projects, some of them were built with MSVC and some were built with MinGW but older versions of the toolchain using sjlj exceptions.

    Would it be safe to build a library (libmpg123 in this case) using a MinGW toolchain which uses Dwarf2 exceptions? Does anybody know which exception model MSVC uses or if either sjlj or dwarf2 are compatible with it?

    Also, when trying to build libmpg123 with the default compiler (mingw32 in the package) it doesn't even pass through configure state failing some tests. Would it be related to dwarf2? Because the other compilers are able to pass through configuration.

    Sorry for the many questions I asked, and excuse any wrong English wording or grammar.

    ReplyDelete
  15. the general assumption is using sjlj for interchange (they are more popular at the moment) and according to some debates is more sane for windows. The configure test errors probably because I messed up mingw headers in 0.5, sorry. I've made corrections maybe tomorrow will be uploaded. I don't know about MSVC's try googling "Structured Exception Handling"

    PS: I'm bad at english too :)

    ReplyDelete
  16. Hi, I wanted to ask if it is possible to change the default GCC toolchain to use when MSYS is opened. I'm using a personal build of it which is in /mingw and I'd like it to be the default one. Is it possible? Thanks.

    ReplyDelete
  17. Yes you can:

    npad /etc/profile

    then edit this line:
    source setgcc mingw32
    with:
    source setgcc mingw

    ReplyDelete
  18. In future can provide in 7z format? the exe decompress is a bit slow, on my 2 core 4 threads system, it only running on single thread...

    ReplyDelete
  19. another stuff, I have to change the following line in msys.bat to work: (add double quote)
    -- set MSSDK=%CurrentInstallFolder%
    ++ set MSSDK="%CurrentInstallFolder%"

    ReplyDelete
  20. @Shin Guey
    in my comp, it felt like the IO throughput is much slower than the actual decompression so I can't really measured it, maybe LZMA2 may help a bit.

    about MSSDK, did you get error in that line or afterward? because the next line will switch to 8.3 dosname if space contained path detected.

    ReplyDelete
  21. I think it would be waaaay better, if you package everything separately, so the user could decide what he needs to have instead of downloading and extracting the whole 4.5gb package... for example mingw-w64 with gcc, g++ for mingw-w64, gdc for mingw-w64, gfortran for mingw-w64, gnat for mingw-w64, harbour for mingw-w64 and etc... what do you think about it?

    ReplyDelete
    Replies
    1. A dilemma I'd say,

      I don't plan on including the extra compilers beyond c/c++ originally, but later realized they are crucial in OSS world regardless the number of users.

      About being individual package or not, the extra compilers part and some supplementary apps is an exception, but in newbie POV, people usually want a complete base installation (my prediction is about 1GB) at the first try that just work. The analogy of "base installation" can be taken from miktex. The decision to compress all in one installer in 0.7 is because otherwise it will exceed the CD size barrier (thanks to 7z solid archive) and because I haven't finished doing cleanups. Strict individual packages (like mingw.org) is very inefficient for 7zip and too intimidating for newbie. Newbie don't need to be bothered with "Do I need this?" or "What's that?" thoughts

      The next one will return to multiple installers, at the moment I think of:
      Base: One shot installer
      Extra: Customizable Installer (extract to pool folder and do selective installation
      Docs: One shot installer (separate from base as some may prefer online manuals)
      All the three should still fit on a CD

      Delete
  22. Hello, Tuma! What do you think of including a cross-compiler in your package? For example, you've already included mingw 4.7.2, and you add gcc 4.7.2 build targetting linux, but on windows. The same stuff for darwin (macos). So that software could be developed on windows operating system and compiled to many different operating systems and architectures (x86 and x64). I think it would be very usefull!

    ReplyDelete
    Replies
    1. Hi thanks for the suggestion

      I did include ARM cross-compiler in 0.6, but later drop it because I don't know how to verify its functionality nor I'm familiar with embedded device similarly while I have hackintosh and opensuse at other partitions I rarely using them. I'm a windows zealot ;p

      OTOH, you could always add as many as gcc installation you want into / directory and switch among them. I think mingw-w64 @ sourceforge has linux targeted gcc and mac/ios? target @ googlecode project (not sure about the later)

      Delete
  23. Hi! You've done a really nice job! There're even languages in your package that I didn't know about, like Harbour. ;)

    Anyway, I found out that liburlmon.a is absent in your mingw-w64 package, it is not really a big deal, I created it with a dlltool, but still.

    Also talking about cross compilation: what do you think about CodeTyphon project? It is a collection of sources of cross compiler tools and libs for FreePascal and Lazarus IDE with a nice GUI interface and minimal binary distribution to bootstrap them. You download archive, extract it and simply compile cross compilers and tools you need with a GUI interface.

    ReplyDelete
    Replies
    1. Thanks,

      For bundled libraries, IMO using a buildscript is preferable to add more stuff beyond the very commonly used.
      If I understand correctly about codetyphon. It is a distro version of lazarus ala gentoo. So basically it against the console oriented philosophy of mine. Things like Lazarus/Qt are very tight suite with established GUI workflow. In my case GUI will be as minimal as possible and zenity will play that role.

      Or maybe you could elaborate what you wish related to CodeTyphon?

      Delete