Tag Archives: Linux

100,000 views and counting

figure5I’m  celebrating the 100,000 views mark on my “Logical Volume Management on #Linux” piece on +developerworks  http://www.ibm.com/developerworks/linux/library/l-lvm2/index.html

Actually, it’s almost at the 200,000 views mark if we consider both Russian and Japanese translations.

And it’s a 7 year article so it’s time for an update or a new article altogether. What would you like to see on the #LVM or #PowerLinux theme? Tweet or send comments!

Volume management is not new in the -ix world (UNIX, AIX, and so forth). And logical volume management (LVM) has been around since Linux kernel 2.4v1 and 2.6.9v2. This article reveals the most useful features of LVM2–a relatively new userspace toolset that provides logical volume management facilities–and suggests several ways to simplify your system administration tasks. Based on reader feedback, the author has updated Listings 10, 14, 15, and…

Jenifer Hopper: Untangling memory access measurements – memory latency

Is memory latency important for you? Are you an optimizer and wanna make sure you’re really extracting all the juice a PowerLinux server can give you?

Or perhaps you simply want to learn more about lmbench, NUMA, Caches and prefetch?

If any of the above is true, make sure you check the excellent article by Jenifer Hopper called “Untangling memory access measurements – memory latency” that was just published on the PowerLinux developerWorks community.

In a very objective, and very well illustrated way, Jenifer explains in two or three pages the important stuff you need to know to start bragging about how you optimized your PowerLinux “Data Stream Control Register” (your Gentoo-building friends will be jealous)

PowerLinux Community, a recent technical piece is “on fire” with more than 2k hits on our PowerLinux …

PowerLinux Community,
a recent technical piece is "on fire" with more than 2k hits on our PowerLinux developerWorks community.
"Taking Advantage of 64-bit DMA capability on PowerLinux" by +Breno Leitao, +Nish Aravamudan, +Bill Buros, +Murilo Vicentini and others talks about taking advantage of special PCIe slots for better IO performance.
Make sure you go check why this article is so popular!

By: Breno Leitão

On new IBM® Power Systems™ servers running Linux, a set of the PCIe slots support a unique feature called 64-bit Direct Memory Access (DMA) , which improves I/O operations.

A new article was written explaining this feature and how to take advantage of it. Moreover, performance data is discussed throughout the article, as comparing the bandwidth and latency improvement provided by this feature when using a 10Gbps Netwo…

Taking Advantage of 64-bit DMA capability on PowerLinux (The PowerLinux Community)

PowerLinux Community,
a recent technical piece is "on fire" with more than 2k hits on our PowerLinux developerWorks community.
"Taking Advantage of 64-bit DMA capability on PowerLinux" by +Breno Leitao, +Nish Aravamudan, +Bill Buros, +Murilo Vicentini and others talks about taking advantage of special PCIe slots for better IO performance.
Make sure you go check why this article is so popular!

By: Breno Leitão

On new IBM® Power Systems™ servers running Linux, a set of the PCIe slots support a unique feature called 64-bit Direct Memory Access (DMA) , which improves I/O operations.

A new article was written explaining this feature and how to take advantage of it. Moreover, performance data is discussed throughout the article, as comparing the bandwidth and latency improvement provided by this feature when using a 10Gbps Netwo…

I recommend any IT professionals and enthusiasts to check out what “PureFlex System Manager” is all about…

I recommend any IT professionals and enthusiasts to check out what "PureFlex System Manager" is all about, in one of the webinars listed below.

IBM PureFlex™ System combines compute, storage, networking, virtualization and management into a single infrastructure system. An integral feature of the Pu

Recap of IBM Linux and Open-Source thought leaders participation of major conferences: – April 29-30…

Recap of IBM Linux and Open-Source thought leaders participation of major conferences:

– April 29-30 Computerworld Open Source Business Conference (OSBC)
https://www.eiseverywhere.com/ehome/50794/87762/
Jim Wasko on a Cloud panel, Gord Sissons on Hadoop and PowerLinux, Jean Staten-Healy on KVM and OpenStack panel.

– May 14-15 LF's Enterprise End User Summit
http://events.linuxfoundation.org/events/enterprise-end-user-summit/schedule
Jean Staten-Healy and Mike Day in two different talks about KVM, Dan Frye, our Open Systems Development VP talks about KVM on the Enteprise, Wilhelm Mild on Linux on System z, Paul McKenney on the kernel development panel, Sean Dague on Open Cloud, and Paul McKenney also talks about Bare Metal Performance

PowerLinux servers are also available for direct remote access through our Virtual Loaner Program..

PowerLinux servers are also available for direct remote access through our Virtual Loaner Program..

IBM VLP Virtual Loaner Program: Remote access to IBM hardware, software, and storage

Understanding the Java performance improvements seen on POWER7+ (The PowerLinux Community)

By Mauricio Oliveira Remarkable Java performance improvements are being seen on the latest IBM Power Systems featuring POWER7+ processors which have significantly bigger, 10MB L3 caches associated w…

Exporting GIT repositories remotely

This tip may be useful for those working on a local GIT tree, but in need to “export” it to a remote server (for example for building and testing).

I know that ideally one would “push” the changes to a commonly-accessible remote server, and pull back from the build/testing host. But sometimes we simply don’t have GIT available on that host, other times we’re just lazy.

The standard procedure then is to use the “git archive” command to export the contents of the repository to a tarball, optionally compress it, and then transfer it and unpack on the remote side:

$ git archive --format=tar --prefix=kernel_20101221a/ HEAD | bzip2 > kernel_20101221a.tar.bz2
$ scp kernel_20101221a.tar.bz2 root@spin.ltc.br.ibm.com:.
root@spin.ltc.br.ibm.com's password:
kernel_20101221a.tar.bz2                                                  100%   68MB  11.4MB/s   00:06    
$ ssh root@spin.ltc.br.ibm.com
root@spin.ltc.br.ibm.com's password:
Last login: Tue Dec 21 15:01:59 2010 from dyn531363.br.ibm.com
# tar xvf kernel_20101221a.tar.bz2
kernel_20101221a/
kernel_20101221a/.gitignore
kernel_20101221a/.mailmap
kernel_20101221a/COPYING
kernel_20101221a/CREDITS
kernel_20101221a/Documentation/
kernel_20101221a/Documentation/.gitignore
kernel_20101221a/Documentation/00-INDEX
kernel_20101221a/Documentation/ABI/
...

This has obvious problems… Even if you’re using SSH key authentication to avoid entering the password twice, you end-up typing too many commands and transferring too many bytes. For a project as large as the Linux kernel, this could be a real pain.

A smarter, more direct alternative would be to avoid creating a local archive at all, using ssh input/output tunneling and tar to do the hard-lifting for us:

$ git archive --format=tar --prefix=kernel_20101221b/ HEAD | ssh root@spin.ltc.br.ibm.com tar xvf -
root@spin.ltc.br.ibm.com's password:
kernel_20101221b/
kernel_20101221b/.gitignore
kernel_20101221b/.mailmap
kernel_20101221b/COPYING
kernel_20101221b/CREDITS
kernel_20101221b/Documentation/
kernel_20101221b/Documentation/.gitignore
kernel_20101221b/Documentation/00-INDEX
kernel_20101221b/Documentation/ABI/
kernel_20101221b/Documentation/ABI/README
...

Since you may end-up doing this several times per day, an even smarter way would be to just transfer the absolute necessary – essentially what changed between the last and the current tree. RSYNC is the ideal tool to do that, since it will only send the differences between those files. We could, in theory, use rsync to transfer the whole tree, perhaps ignoring the .git directory since it has no use for us remotely. An even better option would be to transfer just what’s being tracked (and rsync would transfer just what changed). Doing that requires us to use “git ls-files” to list the files being tracked, and pipe that to a rsync command that reads the files to be transfered from standard input:

$ git ls-files -z | rsync -e ssh --files-from - -av0 . root@spin.ltc.br.ibm.com:kernel_20101221c/
root@spin.ltc.br.ibm.com's password:
building file list ... done
Documentation/
Documentation/ABI/
Documentation/ABI/obsolete/
Documentation/ABI/removed/
Documentation/ABI/stable/
Documentation/ABI/testing/
Documentation/DocBook/
Documentation/DocBook/drm.tmpl
Documentation/DocBook/filesystems.tmpl
Documentation/DocBook/gadget.tmpl
Documentation/DocBook/genericirq.tmpl
Documentation/DocBook/kernel-api.tmpl
Documentation/DocBook/kernel-hacking.tmpl
Documentation/DocBook/kernel-locking.tmpl
Documentation/DocBook/kgdb.tmpl
Documentation/DocBook/libata.tmpl
Documentation/DocBook/librs.tmpl
Documentation/DocBook/lsm.tmpl
Documentation/DocBook/mcabook.tmpl
Documentation/DocBook/media-entities.tmpl
Documentation/DocBook/media-indices.tmpl
Documentation/DocBook/media.tmpl
...

The “git ls-files” command will list only the files being tracked in the current git tree. The “-z” argument for “git ls-files”, together with it’s counterpart “-0” in the rsync command tell those commands to use “\0” (the null character) as delimiter between files, so that is safe to deal with file names with spaces on them.

The final step is to create an alias that will invoke the command above:

$ git config --add alias.upload-spin '!git ls-files -z | rsync -e ssh --files-from - -av0 . root@spin.ltc.br.ibm.com:kernel-dev/'
$ git upload-spin
root@spin.ltc.br.ibm.com's password:
building file list ... done
created directory kernel-dev

Note that “shell” aliases (i.e., starting with “!”) will execute those commands on the top-level directory for the GIT repository you are on, so the command above should work correctly even from sub-directories.

I hope the above tips can increase your productivity when working with cross-platform development (and more important, freeing you from a boring repetitive task to more do more value-add coding).

Leave a comment if you like it, have corrections or would like to show us some other tips.

-Klaus

Update:

Sometimes you’ll need a clean upload, meaning you’d like to remove untracked files from the remote side. Turns out this is not as easy as I have hoped.

I’m using the the output from “git ls-files” as an “inclusion filter”, plus an “include all dirs” and a “exclude everything” filter to the rsync command. The reasons why I’m using those are beyond the scope of this post (check the INCLUDE/EXCLUDE PATTERN RULES section at the rsync(1) man page), but the command below seems do to the trick:

git ls-files -z | rsync -avi0 -e ssh --include-from - --include '*/' --prune-empty-dirs \
    --delete --delete-excluded --exclude '*' . root@spin.ltc.br.ibm.com:kernel-dev/

I’ve created an alias called “update-pristine” that does that automatically for me. It will take more time to execute than the original version (I believe due to recursive path descend), but again, you should only use it when wanting to explicitly exclude everything that is not being tracked.