Category Archives: SysAdmin

Bash on Windows 10

I was pleasantly surprised to see that there is beta support for a Ubuntu-based bash on Windows 10.

An introduction is here, along with an Installation Guide on the MSDN website.

apt-get pulls an obsolete version of NodeJS. To load the latest (as of today) NodeJS, use the following commands with root privileges:

apt-get -y update
apt-get install git
curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -
apt-get install -y nodejs

Install NodeJS and MongoDB on CentOS 7

As always, I’m sharing my personal notes here. I’ve used virtual machines for development since the first version of VMware. I’ve been using DigitalOcean’s services for some time. I generally use their CentOS 7 install.

NodeJS

Update: I have newer install instructions here.

Edit: Removed old instructions.

Some native add-ons from npm require build tools. If you anticipate needing those, install:

yum -y install gcc-c++ make

MongoDB

Installing MongoDB’s community edition requires creating a yum repo file, a a system configuration file, and then a reboot.

vim /etc/yum.repos.d/mongodb-org-3.2.repo

and fill it with the following text:

[mongodb-org-3.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.2/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.2.asc

which allows MongoDB to be installed with a simple:

yum install -y mongodb-org
service mongod start
chkconfig mongod on

Getting Rid of Warnings

If you use the mongo client, you’ll notice the following warnings:

Server has startup warnings:
2016-09-04T18:27:20.393+0000 I CONTROL  [initandlisten]
2016-09-04T18:27:20.393+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2016-09-04T18:27:20.393+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-09-04T18:27:20.393+0000 I CONTROL  [initandlisten]
2016-09-04T18:27:20.393+0000 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2016-09-04T18:27:20.393+0000 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2016-09-04T18:27:20.393+0000 I CONTROL  [initandlisten]
2016-09-04T18:27:20.393+0000 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 4096 processes, 64000 files. Number of processes should be at least 32000 : 0.5 times number of files.
2016-09-04T18:27:20.393+0000 I CONTROL  [initandlisten]

The soft rlimits warning can be fixed by creating a file:

 vim /etc/security/limits.d/90-nproc.conf

and putting the following line in it:

mongod     soft    nproc     64000

and rebooting the system.

The transparent_hugepage warnings can be fixed by creating a file:

vim /etc/init.d/disable-transparent-hugepages

and filling it with:

#!/bin/bash
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    re='^[0-1]+$'
    if [[ $(cat ${thp_path}/khugepaged/defrag) =~ $re ]]
    then
      # RHEL 7
      echo 0  > ${thp_path}/khugepaged/defrag
    else
      # RHEL 6
      echo 'no' > ${thp_path}/khugepaged/defrag
    fi

    unset re
    unset thp_path
    ;;
esac

ensuring it runs at startup:

chmod 755 /etc/init.d/disable-transparent-hugepages
chkconfig --add disable-transparent-hugepages

and then, finally, rebooting. The mongo client should start without warnings now:

 
MongoDB shell version: 3.2.9
connecting to: test
>

 

 

Resolving the git error “pack exceeds maximum allowed size” during push

Elephant_near_ndutuI have a large repository that takes up a modest number of gigabytes. When attempting to push it to a  new remote repository,  the push failed, complaining that the pack size exceeds the maximum allowed.

First of all, let’s get out of the way the fact that repacking the local repository or fiddling with the pack.packSizeLimit limit configuration setting won’t fix the problem. That will simply tidy  up your local machine.

As I understand it (corrections welcome), the problem is a collision of several things. When performing this massive beginning-to-end push, git creates a massive pack on  the fly and pipes that across the network to the remote machine. The remote machine needs  to be able to perform memory mapping on this huge wad of data. File system, CPU architecture, and memory needs have to be satisfied for this to work. Otherwise, the pack size error is reported and the push fails. Annoyingly, this can happen after you’ve transferred gigabytes of data across a network with a bottleneck, completely wasting a lot of time.

Fortunately the work-around is simple. Push the repository in chunks, working your way up the tree.

If your repository has a lot of branching, you may be able to push a branch at a time, as the generated pack will be for that branch.

This repo of mine has a very linear history, and feeling a little lazy I used my git GUI (SourceTree) to make a temporary branch about a third way up the tree, and pushed that. I moved the temporary branch another third of the way up the tree, and pushed that. Finally I could push master and remove the temporary branch.

If the repository were big and hairy enough one could write a script to traverse the tree and programmatically push at appropriate commit points, but for me it’s an exceptional situation that doesn’t warrant that type of effort.

 

Control VMware Fusion from the Command Line

Background

Virtualization has been a key technology for me as a contractor. Because I have multiple clients, keeping their projects sequestered from each other is a snap: one or more VMs (virtual machines) per project. Development, testing, and in some cases deployment is all on separate VMs.

I’ve been using VMware Workstation since it first came out 1999. The ability for me to seamlessly move VMs between Windows, Linux, and OS X has been a key reason I’ve stuck with them for so long.

Automating tasks via scripting one of many reasons I insist that my workstation run  Un*x of some flavour. I’ve been using OS X for quite a while quite happily as a software developer, however VMware Fusion (for OS X) doesn’t present the full set of features available on VMware Workstation (for Linux and Windows). Fortunately hidden under the hood are the same tools.

The Executable vmrun

VMware provides a tool called vmrun that allows common operations to be performed on VMs — starting, suspending, taking snapshots, etc. On OS X it’s tucked away in the VMware Fusion bundle under Contents/Library.

In my shell startup I add the directory to PATH:

[bash]
# VMware Fusion
if [ -d "/Applications/VMware Fusion.app/Contents/Library" ]; then
export PATH=$PATH:"/Applications/VMware Fusion.app/Contents/Library"
fi
[/bash]

Find the Virtual Machine’s .vmx File

To use vmrun you need to have the path to the .vmx file that resides inside of the VM bundle on OS X. For example, I have a VM with an install of RedHat Enterprise Linux 7:

RHEL7 VM

This is actually not a single file, but a directory called RHEL7.vmwarevm. The contents can be seen in the Finder by right clicking:

Show Contents

This will open up the directory and show the various files that make up the virtual machine.

Inside the Bundle

As can be seen, the .vmx file is prominently displayed.

Put the Pieces Together

To start the VM from the command line one uses the “start” parameter. For example, if the above virtual machine were in one’s home directory one can type:

[bash]
$ vmrun start ~/RHEL7.vmwarevm/RHEL7.vmx
[/bash]

vmrun Commands

If one runs vmrun without parameters it gives a farily long summary of commands that it accepts. One can:

  1. control the power state of the VM,
  2. control snapshots,
  3. perform various operations inside a running VM, and
  4. other operations such as installing tools and cloning.

A short list of common operations:

Description Command Parameters
List running VMs list
Start a VM start /path/to/vmx/file
Suspend a VM suspend /path/to/vmx/file
Take a snapshot snapshot /path/to/vmx/file snapshot name

 Additional Information

VMware has a PDF on vmrun here.

 

Controlling RHEL 7 Services

One change from RHEL/CentOS 6 to the RHEL 7 beta is how services are controlled. The old service and chkconfig commands are replaced with systemctl. These are my quick and dirty notes compiled from the Fedora Project systemd and SysVinit to Systemd Cheatsheet pages.

Basic Control

The old system command’s replacement is very similar, with services having .service appended:

[bash]
systemctl start|stop|restart|status name.service
[/bash]

For example:

[bash]
systemctl restart httpd.service
[/bash]

Service Boot time Control

To get a list of available services and their boot time status:

[bash]
systemctl list-unit-files –type=service
[/bash]

To set a service to start (or not) at boot time:

[bash]
systemctl enable|disable <em>service</em>.service
[/bash]

For example:

[bash]
systemctl enable mariadb.service
systemctl enable httpd.service
[/bash]

Run Levels

Run levels are called targets, have been simplified, and have names now. An incomplete list:

  1. poweroff.target (run level 0)
  2. rescue.target (single-user mode; run level 1)
  3. multi-user.target (normal run level 3)
  4. graphical.target (normal run level 5)

To set the default run level:

[bash]
systemctl set-default multi-user.target
[/bash]

To change the run level:

[bash]
systemctl isolate name.target
[/bash]

For example, to enter single user mode:

[bash]
systemctl isolate rescue.target
[/bash]

And the appropriate services will be stopped and started.

Additional Reading

  • A description of how systemd fits into the boot process here.
  • Another nice summary here.

Updates

2014-07-17
Updated setting the default run level per CertDepot’s suggestion. Added the “Additional Reading” section.