ソフトウェア構築に関するメモ

LFS システムを構築した皆さんであれば、ソフトウェアのダウンロードと伸張 (解凍) の方法は既にご存知のはずです。 しかしここでは、ソフトウェア構築に不慣れな方に向けてそういった情報も何度か説明することにします。

インストール説明を行っている個々のページでは、パッケージのダウンロード先 URL を示しています。 The patches; however, are stored on the LFS servers and are available via HTTP. These are referenced as needed in the installation instructions.

While you can keep the source files anywhere you like, we assume that you have unpacked the package and changed into the directory created by the unpacking process (the 'build' directory). We also assume you have uncompressed any required patches and they are in the directory immediately above the 'build' directory.

特に明確には述べていませんが、パッケージビルド時は きれいなソースツリー にて作業を進めてください。 configure 処理中やコンパイル中にエラーが発生した場合は、もう一度ビルド作業を進めるなら、いったんソースツリーを削除した上で、パッケージソースの伸張(解凍)からやり直すのが適切なやり方です。 もちろんあなたが独自の Makefile なり C コードなりを用いているような熟練ユーザーであれば話は別ですが、自信がない場合は全くの新しいソースツリーから作業を始めることにしてください。

(root ではない) 一般ユーザーによるソフトウェア構築

Unix システム管理における鉄則は、スーパーユーザーによる操作は必要な時にのみ行うということです。 そこで BLFS でも、ソフトウェアをビルドする際には一般ユーザーにて行い、インストール時のみ root ユーザーとなって作業することとしています。 本書中では、どのパッケージであってもこのやり方で進めます。 特別に指定されていない限りは、すべての手順を一般ユーザーにて実施していきます。 必要な時には root 権限にて作業を進めるべきであることも説明します。

ソフトウェアの伸張 (解凍)

ファイルが .tar 形式でかつ圧縮されている場合は、以下のいずれかのコマンドにより伸張 (解凍) することができます。

tar -xvf filename.tar.gz
tar -xvf filename.tgz
tar -xvf filename.tar.Z
tar -xvf filename.tar.bz2
[注記]

注記

上に示すコマンドや、これ以降に示すコマンドにおいても v パラメーターはつけなくても構いません。 これをつけないようにすれば、アーカイブから抽出されるファイル一覧の表示が省略されます。 抽出処理時間が短縮されて、抽出中にエラーが発生した場合には判別しやすくなります。

あるいは以下のようなやり方もあります。

bzcat filename.tar.bz2 | tar -xv

Finally, you sometimes need to be able to unpack patches which are generally not in .tar format. The best way to do this is to copy the patch file to the parent of the 'build' directory and then run one of the following commands depending on whether the file is a .gz or .bz2 file:

gunzip -v patchname.gz
bunzip2 -v patchname.bz2

'md5sum' を使ったファイルの整合確認

Generally, to verify that the downloaded file is genuine and complete, many package maintainers also distribute md5sums of the files. To verify the md5sum of the downloaded files, download both the file and the corresponding md5sum file to the same directory (preferably from different on-line locations), and (assuming file.md5sum is the md5sum file downloaded) run the following command:

md5sum -c file.md5sum

If there are any errors, they will be reported. Note that the BLFS book includes md5sums for all the source files also. To use the BLFS supplied md5sums, you can create a file.md5sum (place the md5sum data and the exact name of the downloaded file on the same line of a file, separated by white space) and run the command shown above. Alternately, simply run the command shown below and compare the output to the md5sum data shown in the BLFS book.

md5sum <name_of_downloaded_file>

インストール中のログファイル生成

For larger packages, it is convenient to create log files instead of staring at the screen hoping to catch a particular error or warning. Log files are also useful for debugging and keeping records. The following command allows you to create an installation log. Replace <command> with the command you intend to execute.

( <command> 2>&1 | tee compile.log && exit $PIPESTATUS )

2>&1 redirects error messages to the same location as standard output. The tee command allows viewing of the output while logging the results to a file. The parentheses around the command run the entire command in a subshell and finally the exit $PIPESTATUS command ensures the result of the <command> is returned as the result and not the result of the tee command.

Using Multiple Processors

For many modern systems with multiple processors (or cores) the compilation time for a package can be reduced by performing a "parallel make" by either setting an environment variable or telling the make program how many processors are available. For instance, a Core2Duo can support two simultaneous processes with:

export MAKEFLAGS='-j2'

or just building with:

make -j2

Generally the number of processes should not exceed the number of cores supported by the CPU. To list the processors on your system, issue: grep processor /proc/cpuinfo.

In some cases, using multiple processors may result in a 'race' condition where the success of the build depends on the order of the commands run by the make program. For instance, if an executable needs File A and File B, attempting to link the program before one of the dependent components is available will result in a failure. This condition usually arises because the upstream developer has not properly designated all the prerequisites needed to accomplish a step in the Makefile.

If this occurs, the best way to proceed is to drop back to a single processor build. Adding '-j1' to a make command will override the similar setting in the MAKEFLAGS environment variable.

[注記]

注記

When running the package tests or the install portion of the package build process, we do not recommend using an option greater than '-j1' unless specified otherwise. The installation procedures or checks have not been validated using parallel procedures and may fail with issues that are difficult to debug.

ビルド作業の自動化

There are times when automating the building of a package can come in handy. Everyone has their own reasons for wanting to automate building, and everyone goes about it in their own way. Creating Makefiles, Bash scripts, Perl scripts or simply a list of commands used to cut and paste are just some of the methods you can use to automate building BLFS packages. Detailing how and providing examples of the many ways you can automate the building of packages is beyond the scope of this section. This section will expose you to using file redirection and the yes command to help provide ideas on how to automate your builds.

File Redirection to Automate Input

You will find times throughout your BLFS journey when you will come across a package that has a command prompting you for information. This information might be configuration details, a directory path, or a response to a license agreement. This can present a challenge to automate the building of that package. Occasionally, you will be prompted for different information in a series of questions. One method to automate this type of scenario requires putting the desired responses in a file and using redirection so that the program uses the data in the file as the answers to the questions.

Building the CUPS package is a good example of how redirecting a file as input to prompts can help you automate the build. If you run the test suite, you are asked to respond to a series of questions regarding the type of test to run and if you have any auxiliary programs the test can use. You can create a file with your responses, one response per line, and use a command similar to the one shown below to automate running the test suite:

make check < ../cups-1.1.23-testsuite_parms

This effectively makes the test suite use the responses in the file as the input to the questions. Occasionally you may end up doing a bit of trial and error determining the exact format of your input file for some things, but once figured out and documented you can use this to automate building the package.

yes を使った入力の自動化

入力プロンプトに対して決まった内容を入力したり、それが複数回あってもすべて同一の答えを入力するような場合があります。 そういった時は yes コマンドを利用すると便利です。 yes コマンドは、何度かある問合せ入力に対して同一の答えを入力するものです。 入力内容として、単に Enter キーを入力する、Y キーを入力する、所定の文字列を入力する、といったことが可能です。 単純な利用例を以下に示します。

初めに以下のコマンドを実行して Bash スクリプトを生成します。

cat > blfs-yes-test1 << "EOF"
#!/bin/bash

echo -n -e "\n\nPlease type something (or nothing) and press Enter ---> "

read A_STRING

if test "$A_STRING" = ""; then A_STRING="Just the Enter key was pressed"
else A_STRING="You entered '$A_STRING'"
fi

echo -e "\n\n$A_STRING\n\n"
EOF
chmod 755 blfs-yes-test1

まずはこのスクリプト ./blfs-yes-test1 をコマンドラインから実行してみます。 入力が促されて処理が止まるので、何かを入力して (あるいは何も入力せずに) Enter キーを入力します。 入力した内容は画面上に表示されます。 さてそこで yes コマンドを用い、入力を自動化することにします。

yes | ./blfs-yes-test1

この場合、yes のパイプ処理を通じて、スクリプトに対しては y が入力されたものとして受け渡されます。 以下は特定の文字列を受け渡すような例です。

yes 'This is some text' | ./blfs-yes-test1

The exact string was used as the response to the script. Finally, try it using an empty (null) string:

yes '' | ./blfs-yes-test1

Notice this results in passing just the press of the Enter key to the script. This is useful for times when the default answer to the prompt is sufficient. This syntax is used in the Net-tools instructions to accept all the defaults to the many prompts during the configuration step. You may now remove the test script, if desired.

File Redirection to Automate Output

In order to automate the building of some packages, especially those that require you to read a license agreement one page at a time, requires using a method that avoids having to press a key to display each page. Redirecting the output to a file can be used in these instances to assist with the automation. The previous section on this page touched on creating log files of the build output. The redirection method shown there used the tee command to redirect output to a file while also displaying the output to the screen. Here, the output will only be sent to a file.

Again, the easiest way to demonstrate the technique is to show an example. First, issue the command:

ls -l /usr/bin | more

Of course, you'll be required to view the output one page at a time because the more filter was used. Now try the same command, but this time redirect the output to a file. The special file /dev/null can be used instead of the filename shown, but you will have no log file to examine:

ls -l /usr/bin | more > redirect_test.log 2>&1

Notice that this time the command immediately returned to the shell prompt without having to page through the output. You may now remove the log file.

The last example will use the yes command in combination with output redirection to bypass having to page through the output and then provide a y to a prompt. This technique could be used in instances when otherwise you would have to page through the output of a file (such as a license agreement) and then answer the question of do you accept the above?. For this example, another short Bash script is required:

cat > blfs-yes-test2 << "EOF"
#!/bin/bash

ls -l /usr/bin | more

echo -n -e "\n\nDid you enjoy reading this? (y,n) "

read A_STRING

if test "$A_STRING" = "y"; then A_STRING="You entered the 'y' key"
else A_STRING="You did NOT enter the 'y' key"
fi

echo -e "\n\n$A_STRING\n\n"
EOF
chmod 755 blfs-yes-test2

This script can be used to simulate a program that requires you to read a license agreement, then respond appropriately to accept the agreement before the program will install anything. First, run the script without any automation techniques by issuing ./blfs-yes-test2.

Now issue the following command which uses two automation techniques, making it suitable for use in an automated build script:

yes | ./blfs-yes-test2 > blfs-yes-test2.log 2>&1

If desired, issue tail blfs-yes-test2.log to see the end of the paged output, and confirmation that y was passed through to the script. Once satisfied that it works as it should, you may remove the script and log file.

Finally, keep in mind that there are many ways to automate and/or script the build commands. There is not a single correct way to do it. Your imagination is the only limit.

依存パッケージ

本書が示す各パッケージの説明においては、依存するパッケージを一覧表示しています。 その一覧では以下に示すような項目分けを行っています。

  • 必須 は、パッケージを初めてインストールする際には、依存しているそれらのパッケージがない状態では正しくビルドすることができないことを表します。

  • Recommended means that BLFS strongly suggests this package is installed first for a clean and trouble-free build, that won't have issues either during the build process, or at run-time. The instructions in the book assume these packages are installed. Some changes or workarounds may be required if these packages are not installed.

  • 任意 は、付加的な機能を実現するためにはそのパッケージが必要であることを表します。 Often BLFS will describe the dependency to explain the added functionality that will result.

最新のパッケージソースの利用

本書内のパッケージをビルドしようとした際に、ビルド出来なかったり正常に動作しなかったりすることが発生するかもしれません。 本書の編集者は、各パッケージが正常にビルド出来、正常に動作するように常に確認を行っています。 しかしパッケージの確認に見落としがあったり、BLFS の特定バージョンでのテスト確認が不十分であったりするものもあります。

パッケージのビルド不備や動作不備に気づいた方は、そのパッケージのより新しいバージョンが存在しているかどうかを確認してください。 これはつまり、パッケージ管理サイトを参照して最新バージョンを入手し、そのバージョンでのビルドを試して頂きたいのです。 パッケージのダウンロード URL だけでは管理サイトが見つけられなかった場合は、Google を利用してパッケージ名を検索してください。 例えば Google にて 'パッケージ名 download' (引用符は除きます) といった検索語を入力してみてください。 あるいは 'パッケージ名 home page' と入力するのが良いかもしれません。 そうすることでパッケージ管理サイトが見つけ出せるはずです。

Stripping One More Time

In LFS, stripping of debugging symbols was discussed a couple of times. When building BLFS packages, there are generally no special instructions that discuss stripping again. It is probably not a good idea to strip an executable or a library while it is in use, so exiting any windowing environment is a good idea. Then you can do:

find /{,usr/}{bin,lib,sbin} \
    -type f \( -name \*.so* -a ! -name \*dbg \) \
    -exec strip --strip-unneeded {} \;

If you install programs in other directories such as /opt or /usr/local, you may want to strip the files there too.

For more information on stripping, see http://www.technovelty.org/linux/stripping-shared-libraries.html.

Working with different build systems

There are now three different build systems in common use for converting C or C++ source code into compiled programs or libraries and their details (particularly, finding out about available options and their default values) differ. It may be easiest to understand the issues caused by some choices (typically slow execution or unexpected use of, or omission of, optimizatons) by starting with the CFLAGS and CXXFLAGS environment variables. There are also some programs which use rust.

Most LFS and BLFS builders are probably aware of the basics of CFLAGS and CXXFLAGS for altering how a program is compiled. Typically, some form of optimization is used by upstream developers (-O2 or -O3), sometimes with the creation of debug symbols (-g), as defaults.

If there are contradictory flags (e.g. multiple different -O values), the last value will be used. Sometimes this means that flags specified in environment variables will be picked up before values hardcoded in the Makefile, and therefore ignored. For example, where a user specifies '-O2' and that is followed by '-O3' the build will use '-O3'.

There are various other things which can be passed in CFLAGS or CXXFLAGS, such as forcing compilation for a specific microarchitecture (e.g. -march=amdfam10, -march=native) or specifying a specific standard for C or C++ (-std=c++17 for example). But one thing which has now come to light is that programmers might include debug assertions in their code, expecting them to be disabled in releases by using -DNDEBUG. Specifically, if Mesa-20.1.5 is built with these assertions enabled, some activities such as loading levels of games can take extremely long times, even on high-class video cards.

Autotools with Make

This combination is often described as 'CMMI' (configure, make, make install) and is used here to also cover the few packages which have a configure script that is not generated by autotools.

Sometimes running ./configure --help will produce useful options about switches which might be used. At other times, after looking at the output from configure you may need to look at the details of the script to find out what it was actually searching for.

Many configure scripts will pick up any CFLAGS or CXXFLAGS from the environment, but CMMI packages vary about how these will be mixed with any flags which would otherwise be used (variously: ignored, used to replace the programmer's suggestion, used before the programmer's suggestion, or used after the programmer's suggestion).

In most CMMI packages, running 'make' will list each command and run it, interspersed with any warnings. But some packages try to be 'silent' and only show which file they are compiling or linking instead of showing the command line. If you need to inspect the command, either because of an error, or just to see what options and flags are being used, adding 'V=1' to the make invocation may help.

CMake

CMake works in a very different way, and it has two backends which can be used on BLFS: 'make' and 'ninja'. The default backend is make, but ninja can be faster on large packages with multiple processors. To use ninja, specify '-G Ninja' in the cmake command. However, there are some packages which create fatal errors in their ninja files but build successfully using the default of Unix Makefiles.

The hardest part of using CMake is knowing what options you might wish to specify. The only way to get a list of what the package knows about is to run cmake -LAH and look at the output for that default configuration.

Perhaps the most-important thing about CMake is that it has a variety of CMAKE_BUILD_TYPE values, and these affect the flags. The default is that this is not set and no flags are generated. Any CFLAGS or CXXFLAGS in the environment will be used. If the programmer has coded any debug assertions, those will be enabled unless -DNDEBUG is used. The following CMAKE_BUILD_TYPE values will generate the flags shown, and these will come after any flags in the environment and therefore take precedence.

  • Debug : '-g'

  • Release : '-O3 -DNDEBUG'

  • RelWithDebInfo : '-O2 -g -DNDEBUG'

  • MinSizeRel : '-Os -DNDEBUG'

CMake tries to produce quiet builds. To see the details of the commands which are being run, use 'make VERBOSE=1' or 'ninja -v'.

Meson

Meson has some similarities to CMake, but many differences. To get details of the defines that you may wish to change you can look at meson_options.txt which is usually in the top-level directory.

If you have already configured the package by running meson and now wish to change one or more settings, you can either remove the build directory, recreate it, and use the altered options, or within the build directory run meson configure, e.g. to set an option:

meson configure -D<some_option>=true

If you do that, the file meson-private/cmd_line.txt will show the last commands which were used.

Meson provides the following buildtype values, and the flags they enable come after any flags supplied in the environment and therefore take precedence.

  • plain : no added flags. This is for distributors to supply their own CLFAGS, CXXFLAGS and LDFLAGS. There is no obvious reason to use this in BLFS.

  • debug : '-g'

  • debugoptimized : '-O2 -g' - this is the default if nothing is specified, it leaves assertions enabled.

  • release : '-O3 -DNDEBUG' (but occasionally a package will force -O2 here)

Although the 'release' buildtype is described as enabling -DNDEBUG, and all CMake Release builds pass that, it has so far only been observed (in verbose builds) for Mesa-20.1.5. That suggests that it might only be used when there are debug assertions present.

The -DNDEBUG flag can also be provided by passing -Db_ndebug=true.

To see the details of the commands which are being run in a package using meson, use 'ninja -v'.

Rustc and Cargo

Most released rustc programs are provided as crates (source tarballs) which will query a server to check current versions of dependencies and then download them as necessary. These packages are built using cargo --release. In theory, you can manipulate the RUSTFLAGS to change the optimize-level (default is 3, like -O3, e.g. -Copt-level=3) or to force it to build for the machine it is being compiled on, using -Ctarget-cpu=native but in practice this seems to make no significant difference.

If you find an interesting rustc program which is only provided as unpackaged source, you should at least specify RUSTFLAGS=-Copt-level=2 otherwise it will do an unoptimized compile with debug info and run much slower.

The rust developers seem to assume that everyone will compile on a machine dedicated to producing builds, so by default all CPUs are used. This can often be worked around, either by exporting CARGO_BUILD_JOBS=<N> or passing --jobs <N> to cargo. For compiling rustc itself, specifying --jobs <N> on invocations of x.py (together with the CARGO_BUILD_JOBS environment variable, which looks like a "belt and braces" approach but seems to be necessary) mostly works. The exception is running the tests when building rustc, some of them will nevertheless use all online CPUs, at least as of rustc-1.42.0.

Optimizing the build

Many people will prefer to optimize compiles as they see fit, by providing CFLAGS or CXXFLAGS. For an introduction to the options available with gcc and g++ see https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html and https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html and info gcc.

Some packages default to '-O2 -g', others to '-O3 -g', and if CFLAGS or CXXFLAGS are supplied they might be added to the package's defaults, replace the package's defaults, or even be ignored. There are details on some desktop packages which were mostly current in April 2019 at http://www.linuxfromscratch.org/~ken/tuning/ - in particular, README.txt, tuning-1-packages-and-notes.txt, and tuning-notes-2B.txt. The particular thing to remember is that if you want to try some of the more interesting flags you may need to force verbose builds to confirm what is being used.

Clearly, if you are optimizing your own program you can spend time to profile it and perhaps recode some of it if it is too slow. But for building a whole system that approach is impractical. In general, -O3 usually produces faster programs than -O2. Specifying -march=native is also beneficial, but means that you cannot move the binaries to an incompatible machine - this can also apply to newer machines, not just to older machines. For example programs compiled for 'amdfam10' run on old Phenoms, Kaveris, and Ryzens : but programs compiled for a Kaveri will not run on a Ryzen because certain op-codes are not present. Similarly, if you build for a Haswell not everything will run on a SandyBridge.

There are also various other options which some people claim are beneficial. At worst, you get to recompile and test, and then discover that in your usage the options do not provide a benefit.

If building Perl or Python modules, or Qt packages which use qmake, in general the CFLAGS and CXXFLAGS used are those which were used by those 'parent' packages.

Options for hardening the build

Even on desktop systems, there are still a lot of exploitable vulnerabilities. For many of these, the attack comes via javascript in a browser. Often, a series of vulnerabilities are used to gain access to data (or sometimes to pwn, i.e. own, the machine and install rootkits). Most commercial distros will apply various hardening measures.

For hardening options which are reasonably cheap, there is some discussion in the 'tuning' link above (occasionally, one or more of these options might be inappropriate for a package). These options are -D_FORTIFY_SOURCE=2, -fstack-protector=strong, and (for C++) -D_GLIBCXX_ASSERTIONS. On modern machines these should only have a little impact on how fast things run, and often they will not be noticeable.

In the past, there was Hardened LFS where gcc (a much older version) was forced to use hardening (with options to turn some of it off on a per-package basis. What is being covered here is different - first you have to make sure that the package is indeed using your added flags and not over-riding them.

The main distros use much more, such as RELRO (Relocation Read Only) and perhaps -fstack-clash-protection. You may also encounter the so-called 'userspace retpoline' (-mindirect-branch=thunk etc.) which is the equivalent of the spectre mitigations applied to the linux kernel in late 2018). The kernel mitigations caused a lot of complaints about lost performance, if you have a production server you might wish to consider testing that, along with the other available options, to see if performance is still sufficient.

Whilst gcc has many hardening options, clang/LLVM's strengths lie elsewhere. Some options which gcc provides are said to be less effective in clang/LLVM, others are not available.

最終更新日: 2020-07-01 02:45:38 +0900