I am starting to grow tired of Emacs, and enjoyed Atom/LightTable and want to start using them more often (aside from running them essentially sandboxed from the rest of my system). However, using Atom/LightTable, the default mode when “building from source” is to download a pre-built Electron binary. Building Electron from source, however, still downloads precompiled versions of libchromiumcontent (the library version of Chromium content engine). You can set a flag to build libchromiumcontent, but this downloads prebuilt clang binaries from Google unless you set another flag. Ugh.

I prefer to build and package things from scratch, as opposed to downloading precompiled blobs from 3rd parties. This is usually to make sure the build is optimized for my system, and will play nicely with the other packages. Ideally, I think this helps with reproducible builds. I am wary the precompiled binary may have not-so-nice things able to be inserted, and building from source and comparing binary outputs would allow one to compare and make sure it is not compromised. The prebuilt binary leaves GitHub as a single point of failure.

Anyways! This is a log of my attempts to do so.

Compiled Instructions (updated as going along)

# Use Python 2
find . -name '*.py' -exec sed -i -r 's|/usr/bin/python$|&2|g' {} +

# There are still a lot of relative calls which need a workaround
mkdir -p "python2-path"
ln -sf /usr/bin/python2 "python2-path/python"
export PATH="python2-path/python":$PATH

./script/bootstrap.py -v --build_libchromiumcontent --clang_dir /usr/

Log of Progress

First bump, this uses python when needing python2. From the Chromium PKGBUILD, I find/sed’d the python for python2, and then made a temporary sym link of python->python2 and added to PATH.

Initial attempt with ./script/bootstrap.py -v --build_libchromiumcontent --clang_dir /usr/ got stuck when trying to run the build.py script for libchromiumcontent. A quick look shows this is set to a local ninja binary that it downloads and compiles. I would need to change this, possibly adding a new flag to tell it to use a system ninja. As libchromiumcontent also seems to be the lowest level piece in Electron, and based on Chromium, I decided to checkout and build this before continuing more with Electron (with the fringe benefit that I should be able change the “URL” it “downloads” libchromiumcontent from, and avoid more recursive building mess).

There are also messages about it compiling for x64?

Building libchromiumcontent

Compiled

# Download source. In a PKGBUILD, we'd set the pkgver (or a private _chromiumver) and use that variable in the URL. Manually we sub
wget https://commondatastorage.googleapis.com/chromium-browser-official/chromium-51.0.2704.84.tar.xz

# Extract/Rename source
rm -r vendor/chromium
tar -xf chromium-51.0.2704.84.tar.xz
mv chromium-51.0.2704.84 vendor/chromium

# Patch source with chromiumcontent patches
cd vendor/chromium
for file in ../../patches/*.patch; do patch -p1 < $file; done

# Other prepare parts from Arch PKGBUILD
touch vendor/chromium/chrome/test/data/webui/i18n_process_css_test.html

cd ${libchromiumcontent_root}
# Copy Arch/ALARM patches
mkdir patches-alarm
cp -r ${PKGBUILDs}/extra/chromium/*.patch patches-alarm
sed -i \"s/@WIDEVINE_VERSION@/Pinkie Pie/\" patches-alarm/chromium-widevine.patch
cd vendor/chromium
for file in ../../patches-alarm/*.patch; do patch -p1 < $file; done
cd ${libchromiumcontent_root}

# Copy chromiumcontent build files
cp -r chromiumcontent vendor/chromium/

# Set build FLAGS
export MAKEFLAGS=\"-j4\"
export CFLAGS=\"-march=armv7-a -mfloat-abi=hard -mfpu=neon -O2 -pipe -fstack-protector --param=ssp-buffer-size=4 -fno-delete-null-pointer-checks\"
export CXXFLAGS=\"${CFLAGS}\"

# Use Python 2
find . -name '*.py' -exec sed -i -r 's|/usr/bin/python$|&2|g' {} +
mkdir -p \"python2-path\"
ln -sf /usr/bin/python2 \"python2-path/python\"
export PATH=\"$(pwd)/python2-path\":$PATH

# Configure (through either patching or generating gyp files. This is done through a local bash variable array in PKGBUILD, but it just needs to be a string
export CHROMIUM_CONF=\"-Dwerror= -Dclang=0 -Dpython_ver=2.7 -Dlinux_link_gsettings=1 -Dlinux_link_libpci=1 -Dlinux_link_libspeechd=1 -Dlinux_link_pulseaudio=1 -Dlinux_strip_binary=1 -Dlinux_use_bundled_binutils=0 -Dlinux_use_bundled_gold=0 -Dlinux_use_gold_flags=0 -Dicu_use_data_file_flag=1 -Dlogging_like_official_build=1 -Dtracing_like_official_build=1 -Dfieldtrial_testing_like_official_build=1 -Drelease_extra_cflags=\"${CFLAGS} -DUSE_EABI_HARDFLOAT\" -Dlibspeechd_h_prefix=speech-dispatcher/ -Dffmpeg_branding=Chrome -Dproprietary_codecs=1 -Duse_gnome_keyring=0 -Duse_system_bzip2=1 -Duse_system_flac=1 -Duse_system_ffmpeg=1 -Duse_system_harfbuzz=1 -Duse_system_icu=0 -Duse_system_libevent=1 -Duse_system_libjpeg=1 -Duse_system_libpng=1 -Duse_system_libvpx=1 -Duse_system_libxml=0 -Duse_system_snappy=1 -Duse_system_ssl=1 -Duse_system_xdg_utils=1 -Duse_system_yasm=1 -Duse_system_zlib=0 -Dusb_ids_path=/usr/share/hwdata/usb.ids -Duse_mojo=0 -Duse_gconf=0 -Denable_hangout_services_extension=1 -Denable_widevine=1 -Ddisable_fatal_linker_warnings=1 -Ddisable_glibc=1 -Ddisable_sse2=1 -Ddisable_nacl=1 -Ddisable_pnacl=1 -Dsysroot= -Dtarget_arch=arm -Darm_neon=1 -Dv8_use_arm_eabi_hardfloat=true -Darm_float_abi=hard -Darmv7=1 -Darm_fpu=neon\"

# Unbundle deps
# this didn't work and I manually copy-pasted. Will work in PKGBUILD if using array
build/linux/unbundle/replace_gyp_files.py \"${CHROMIUM_CONF[@]}\"

# Configure gyp environment (how chromiumcontent generates the static/shared versions)
## Remove osmesa from chromiumcontent
mg vendor/chromiumcontent/chromiumcontent.gyp

export GYP_GENERATORS=ninja

# generate for static
export GYP_GENERATOR_FLAGS=\"output_dir=out_static_library_arm\"
./build/gyp_chromium --depth=. chromiumcontent/chromiumcontent.gyp \"${CHROMIUM_CONF[@]}\" -Dcomponent=static_library

export GYP_GENERATOR_FLAGS=\"output_dir=out_shared_library_arm\"
./build/gyp_chromium --depth=. chromiumcontent/chromiumcontent.gyp \"${CHROMIUM_CONF[@]}\" -Dcomponent=shared_library

# Ninja build
ninja -C out_static_library_arm/Release chromiumcontent_all
ninja -C out_shared_library_arm/Release chromiumcontent_all

Log

I knew I was likely going to have to do this anyways, as Chromium also will download and build its own versions of dependencies. Building this on its own should allow me to not only instruct it to use system versions of common libraries ("unbundling"), but also to tweak it as need be.

The instructions say building is as easy as calling their helper scripts

./script/bootstrap
./script/update
./script/build

script/bootstrap seems to just init/sync the git submodules. This is easy to do manually or in a PKGBUILD. The only submodules listed are depot_tools and python-patch. In fact, depot_tools is only used for ninja (which I can change to use the system one), so assuming I can obviate python-patch, this will be unnecessary.

script/build is also straightforward: take arguments to trigger target/build-type (ffmpeg or shared/static which is used higher-up in Electron for Release/Debug builds), assemble paths, load configuration, run ninja. This is where it was stuck: it is set to always use the ninja binary in the depot_tools submodule it synced in bootstrap. In fact, the "configuration" is just the directory it outputs to. This can easily be done manually or in PKGBUILD with 1 or 2 ninja calls.

script/update is where things seem complicated… This is at least where it downloads and extracts the tarball for Chromium, patches it, then copies over the libchromiumcontent files.

script/update

This downloads a Chromium source tarball from GitHub (https://github.com/zcbenz/chromium-source-tarball), which describes itself as a synced copy of the official Chromium sources, but with other platform dependencies? It appears to use the same “build environment” that Electron and friends uses (python scripts that automate and setup the build before calling gyp/ninja), which is different than Chromium. However, the scripts seem to checkout Chromium from source using depot_tools, then construct a custom tarball (which it describes as removing excess material the make the tarball smaller). Hopefully this works just like a normal Chromium tree.

It does, but Electron claims there are hundreds of files that need updating for each Chromium release. Is this just the build files? Or does it include glue files and patches? Regardless, libchromiumcontent does use the stock Chromium source, but ties it to a specific version (the one in the VERSION file).

“Copying the chromiumcontent files” are just the gyp project files. I’ll have to pick these apart to make sure they aren’t doing anything too crazy. If not, I can just replace with my own gyp files. Otherwise, I’ll need to patch.

This script calls script/apply-patches which just applies the patches in patchs{,-mas} directories. It uses python-patch, but I don’t see why. My best guess is this is for other platforms, where patch may not exist (or is not portable). Apparently (judging from chromiumcontent.gypi), the -mas patches are for Mac OSX (where does MAS come from then? Just figured it out: Mac Apple Store). These should be able to be safely ignored.

We should apply the Arch/ALARM patches as well. This can easily be done by moving them into the patches directory, so they can be included when I lazily loop through them. However, I think this would be bad technique. The patches in the patch directory would be included in a hash of a source tarball, but the PKGBUILD patches are each hashed, and should be applied individually… meh, laziness wins.

More configuration

The Arch Linux PKGBUILD stores the configuration as an array of compiling macro defines, which it then passes in to gyp_chromium. This array is also used to “unbundle” chromium using “build/linux/unbundle/replace_gyp_files.py”, which quite literally does a file system replacement of the build files for those components with ones that instruct on how to link that library.

This is where things might get complicated. At this point, Arch will call ninja with the “chrome” target after gyp_chromium and be done. Electron, however, uses it’s own gyp files. This seems to be a combination of configuration (seems mostly contained in chromiumcontent.gypi), and the build rules for the shared/static libraries (chromiumcontent.gyp). I’m trying to dissect this gyp file to see if it’s necessary, and if it needs to be changed.

The chromiumcontent.gyp does specify new targets that selectively include chrome dependencies. What is annoying is they list the third-party mesa as a dependency, and this says in that gyp file it’s the software render… This one does not have a “use_system_mesa” flag existing, and I’m trying to walk the “chrome” target dependency tree to see if/when it includes the “osmesa” target of mesa…

OKAY! It seems the third_party mesa is pretty much used in weird edge cases or [mainly] for unittests (which makes sense if these are headless machines it runs tests on).

I’m like 99% sure this is true. The only place that it might leak into the final build is via content_shell (even though, on Linux, this only gets built in tests, or when building “All”). So! I’m taking it out of the chromiumcontent gyp file (and discluding the chromiumcontent.gypi since it sets things we don’t need).

I could not get this to compile on my RPi3. It would get in to the 50th item and get stuck as memory usage shot through the roof. I could see the VFS cache disappear, and once it was gone the system become unresponsive. I dug out my Odroid U3 and it worked like a charm (even though memory usage never got above 500Mb) …until it got to pyproto then failed. Switching to -Duse_system_protobuf=1 got past this, but failed on mksnapshot complaining about invalid instructions: …I forgot the Cortex-A9 only has VFPv3! I just needed to change -mfpu=neon-vfpv4 to -mfpu=neon in CFLAGS.