Frequently Asked Questions

 

FAQ List

1. How can I obtain a copy of the Rowland NMR Toolkit (RNMRTK) program suite?

2. What is shared memory?

3. How do I increase the amount of shared memory on my system?

4. What special changes are needed to make RNMRTK work on an Apple OSX system?

5. Why does the script return "noisecalc: command not found"?

6. Why does the script return "Insufficient memory" when executing the rnmrtk LOAD or LOADVNMR command?

7. My section program does not support the -c and -d options. What do I do?

8. Why do I get an "Illegal size value" error when creating a shared memory section?

9. How can I setup a Varian BioPack sequence to use a sample schedule?

10. How do I invoke rnmrtk?


1. How can I obtain a copy of the Rowland NMR Toolkit (RNMRTK) program suite? (top)

The Rowland NMR Toolkit program suite is made freely available to academic and not-for-profit institutions. It is asked that you fill out a license agreement which can be found here. The license agreement may be mailed to the address listed at the top of the agreement or faxed to 1-860-679-8407. Contact Jeff Hoch for further information about the Toolkit, including commercial licenses.


2. What is shared memory? (top)

Shared memory, as the name implies, is system memory that may be shared between programs. The Rowland NMR Toolkit programs utilize shared memory as a "scratch pad" to store data while computations are performed. The shared memory will utilize physical memory if it is available but can also be located on the disk in a swap partition. Before running RNMRTK programs a shared memory section file needs to be created and needs to be created large enough to contain the data. The size must be large enough to hold the raw NMR data and any signal processing events which may change the data size such as zero-filling. Shared memory sections are created using the RNMRTK program section. By default unix systems have restrictions on the size of shared memory sections that may be created. In order to utilize the RNMRTK it is very likely that you will need to change the configuration files to allow larger shared memory sections to be created. See increasing the size of shared memory sections on my system for more details. Note that the RNMRTK script generator relies on a newer version of the program section which has -c (create) and -d (delete) options. If you have an older version of section which does not support these options it is strongly advised that you obtain a more current version of the software. Contact Jeff Hoch to obtain a more recent version.


3. How do I increase the amount of shared memory on my system? (top)

The maximum size of a shared memory section is controlled by the kernel parameter shmmax. This parameter is set slightly differently depending on the operating system you are using. Below are instructions for Linux, Mac OSX 10.3 and earlier, and Mac OSX 10.4. shmmax may be changed up to a value which is equal to the total amount of memory in the system (physical + swap). Note that increasing the value of shmmax does not affect the amount of available RAM for the system and other programs. Also, once a shared memory section is created with the section program NO memory is actually used. Memory is only used once data is loaded into the shared memory section. Thus creating a shared memory section by itself will do nothing to system performance. However, once data is loaded into the shared memory section it will remain there until it is purged or the shared memory section is destroyed. Note that the script generator by default deletes all shared memory sections at the end of the script to avoid wasting memory by accidentally forgetting to delete the shared memory section after processing is complete. Data can simply be loaded into the RNMRTK program with the load command if needed after the processing is complete.

How large should you set shmmax? You want to set shmmax to a value large enough to handle any data set you may want to process with the RNMRTK. shmmax is entered in bytes. I often set shmmax equal to 1,000 or 2,000 MB on our systems, where 1 MB = 1048576 bytes. The section program, which creates the shared memory sections, creates a 1024 byte header in addition to the size stipulated when it is executed. Because of this I generally set shmmax 1 MB larger than the largest data set size I expect to process. For example if I want to set shmmax to 1 GB the value you will enter for shmmax will be equal to 1048576 x (1000 + 1) = 1049624576.

Information about the current shared memory size limitations for your system may be obtained by using the ipcs command. For linux comptuers you may use ipcs -ml. For OSX systems you may try ipcs -M.


  1. •For Linux computers I add the following lines to the end of the /etc/sysctl.conf file and then reboot. Of course you may choose a different value for shmmax for your system.


# Permit large shared memory sections for use by RNMRTK

# 1048576 x 1001 = 1049624576

kernel.shmmax=1049624576


  1. •If a reboot of the system is not possible you can execute the following command:
    echo '1049624576' > /proc/sys/kernel/shmmax
    Note: This second method will not be permanent and will need to be redone after a reboot.


  2. •For OSX 10.3 computers I edit the following lines in the /etc/rc file and then reboot. Unlike linux computers where shmmax may be set to any value, the value of shmmax should be an integer multiple of the initial value already located in the rc file. For our systems I simply multiplied shmmax by 257 and replaced the value. You also must increase shmall by the same integer multiplier that was used for shmmax (257 in my example). Of course you may choose a different multiplier than 257 for your system.


sysctl -w kern.sysv.shmmax=1077936128

sysctl -w kern.sysv.shmmin=1

sysctl -w kern.sysv.shmmni=32

sysctl -w kern.sysv.shmseg=8

sysctl -w kern.sysv.shmall=263168


  1. •For OSX 10.4 computers I edit the following lines in the /etc/rc file and then reboot. Unlike linux computers where shmmax may be set to any value, the value of shmmax should be an integer multiple of the initial value already located in the rc file. For our systems I simply multiplied shmmax by 257 and replaced the value. You also must increase shmall by the same integer multiplier that was used for shmmax (257 in my example). Of course you may choose a different multiplier than 257 for your system.


# sysctl -w kern.sysv.shmmax=4194304 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=1024

# shmmax and shmall updated for 1024 MB shared memory sections with header by multiplying

# the shmmax and shmall values by 257 from their default values.

sysctl -w kern.sysv.shmmax=1077936128 kern.sysv.shmmin=1 kern.sysv.shmmni=32 kern.sysv.shmseg=8 kern.sysv.shmall=263168


4. What special changes are needed to make RNMRTK work on an Apple OSX system? (top)

Installation of the RNMRTK program suite is not significantly different on an OSX system than any other system. Like all other systems the exe folder of the RNMRTK program should be placed in the path and the maximum size of allowed shared memory sections should be increased (see FAQ #3 for more information). In addition, on OSX systems the user process resource limit for the stack size and the size of the data area  need to be adjusted using the shell built-in command ulimit. To do this add two lines to wherever you set the path. One line should be "ulimit -s 32768" and the other line should be "ulimit -d 300000" both without the quotes. As an example here is a section from my /etc/profile file on an OSX system running bash as my default shell:

# Setup for Rowland NMR Toolkit

PATH=$PATH:/private/imports/cluster-apps/rnmrtk.v3/exe

export PATH

ulimit -s 32768

ulimit -d 300000


5. Why does the script return "noisecalc: command not found? (top)

The program nosiecalc is a new addition to the RNMRTK suite and may not be located in the exe folder of rnmrtk program if you did not recently obtain a copy. The noisecalc program analyzes noise in your spectrum in order to automatically obtain initial guess for def and aim to perform a maximum entropy reconstruction. If noisecal is not in your copy of the RNMRTK you may either go to the downloads and follow the instructions for installing noisecalc, or obtain a more recent version of the RNMRTK program suite. I recommend the latter as there have been many fixes to the program besides the addition of noisecalc.


6. Why does the script return "Insufficient memory" when executing the rnmrtk LOAD or LOADVNMR command? (top)

The RNMRTK script generator is setup to always create the smallest shared memory sections needed to process the data. The RNMRTK program uses 32 bit data (4 bytes) and utilizes a 1024 byte header. When the program section is used to create shared memory sections, only the size of the actual NMR data is entered. The section program automatically multiplies the total size by 4 to deal with the 32 bit data and adds 1024 bytes to take care of the header size. Older version of the section program did not do this. If you have an older version of section program which does not adjust automatically for 32 bit data and the header size you can fix the problem by manually editing the script to make the shared memory sections larger or obtain a more recent version of the RNMRTK program suite. I highly recommend the latter as many other changes to the program have been made. Another possible problem is that there was an earlier error in the script when creating a shared memory section that stated "Illegal value". In this case an existing shared memory section would exist that is too small to hold all of the data and the script encountered an error when trying to make a larger shared memory section. The most likely cause of this would be that the value of shmmax has not been set large enough on your system for the data set you are processing. See FAQ #3 for more information about this.


7. My section program does not support the -c and -d options. What do I do? (top)

The scripts created with the RNMRTK script generator utilize the -c (create) and -d (delete) options of the section program. These options are new additions to the section program and may not be present on your distribution if it is older. The fix is to get the latest version of the RNMRTK program suite.


8. Why do I get an "Illegal size value" error when creating a shared memory section? (top)

The most likely scenario for getting an "Illegal size value" error when creating a shared memory section is that the value of the kernel parameters shmmax or shmall are set too small for the memory section you are attempting to create. shmmax sets the largest single shared memory section which may be created on the system and shmall sets the maximum size of all shared memory sections combined on the system. In order to use the RNMRTK the value of shmmax and possibly shmall will need to be adjusted to a value large enough to hold the NMR data sets which you are processing. See FAQ #3 for instructions on how to do this.


9. How can I setup a Varian BioPack sequence to use a sample schedule? (top)

Forturnatly the folks at Varian and the Orekhov group have made this very easy. Once you have a BioPack sequence loaded and setup properly it is a two step process. First you run the macro BP_NLSinit (which is installed in the latest version of BioPack). This macro creates a new pulse sequence suitable for non-uniform data collection with the same base pulse sequence name with a _S added to the end and compiles it. Creates new parameters needed to use a sample schedule, changes some parameters such as seqfil to match the new sequence, and creates a new parameter list dgnls. By entering dgnls the newly created parameters can be adjusted. To run the BP_NLSinit macro you simply type BP_NLSint(dim) from the VNMR command line, where dim is the number of dimensions in the experiment. For VNMRJ these steps are done through buttons in the Digital Filters page. However, I do not have a copy of VNMRJ yet so I can't speak for the exact method. It should be quite straightforward. The second step is to create a sample schedule and adjust parameters so that the sample schedule is used in the data collection. There are two options for this. One is to let the Varian macros generate the sampling schedule for you based on information provide in the dgnls window. The second is to create your own schedules with sampsched2d, coast, our on-line sample scheduler, or any other method you choose. To use the Varian macro method you need to set nimax and ni2max to the maximum increment size you want in each dimension. You then set ni and ni2 to some values. The product of ni and ni2 will determine the number of acquisition units (lines in the sample schedule) that are created. ni and ni2 lose their traditional meaning when using a sample schedule as both are incremented together based on the values in the sample schedule. They are only used here as a means of setting the number of lines in the sample schedule. There is one issue however, the macro is designed so that ni and ni2 must be smaller than nimax and ni2max, respectively. Once ni and ni2 are set you must set T2sp and T2sp2 which controls the decay constant which shifts the distribution of points that will be picked. The parameter SPARSE should be set to "y" and then you execute the macro BP_NLSset. This will create a sample schedule in the /vnmrsys/exp# folder with a name that is the same as the pulse sequence but has a .hdr_3 extension. If you are using your own sample schedule simply copy the schedule to the /vnmrsys/exp# folder and rename it to have the same name as the pulse sequence but with a .hdr_3 extension. Set ni=1 and ni2=the number of lines in the sample schedule. Make sure the SPARSE flag is set to "y" and you are ready to go. Just a few notes. This procedure describes a 3D case, but can be easily adapted to other dimensionality experiments. This method uses sample schedules that start at 0. The RNMRTK needs the sample schedule to start at 1. In the downloads page you may download a Perl script that can be used to adapt sample schedules by either adding or subtracting 1 from all the values.. Make sure the the SPARSE flag is set to "y" or you will collect a "normal" experiment. Also make sure that the phase and phase2 parameters are set properly (typically to phase2,phase) for 3D experiments.


  1. 10.How do I invoke rnmrtk?    

  2. 1. Locate the RNMRTK executable programs


In order to use the Toolkit you have to know where the programs reside on your computer. Once you know where they are located, the easiest way to use the Toolkit is to add the name of the directory they reside in to your PATH environment variable.
For example, on vp3 the Toolkit resides in /usr3/rnmrtk/rnmrtk.v3/exe. Once you have added the directory name to your PATH, you invoke the Toolkit and its ancillary tools simply by typing their names, e.g., rnmrtk.


[29] setenv PATH ${PATH}:/usr3/rnmrtk/rnmrtk.v3/exe

[30] printenv PATH

:/usr/bsd:/usr/bin:/bin:/usr3/rnmrtk/rnmrtk.v3/exe


  1. 2.Create a shared memory section


The Toolkit needs a "scratch pad" to store the data while it carries out computations. It uses something called a shared memory section to do this. Shared memory resides in the virtual address space of the computer; if there is lots of random access memory available, it will usually reside there, but when there is not enough memory, parts of the shared memory section may by written out to the disk, in an area reserved for use by the operating system called swap space. Shared memory is just that: it can be shared by more than one program at a time. For example, you could run RNMRTK in one window, and run Seepln or Contour in another window to display the results processing by RNMRTK. It is also possible to have more than one window running RNMRTK connected to the same shared memory section; usually this is not a good idea, since it easy to get into conflicts.


Whenever RNMRTK or one of the other programs is invoked, it will tell you if there are other programs attached to the same shared memory section. One additional feature of shared memory sections is worth noting: they are not permanent -- the data they contain will not be saved unless you explicitly tell RNMRTK to save it for you. Shared memory sections are destroyed whenever the system re-boots, but otherwise they linger. They occupy system resources (memory or swap space) even when they are not being used, so it is always good etiquette to remove shared memory sections when you are finished using them.


Before creating a shared memory section, you first need to know how much space you will need. The amount is determined by the size of the spectrum you want to compute. The program used to create a shared memory section, called Section (appropriately enough) prompts you for the size of the spectrum in each dimension. You should give it large enough numbers so that the spectrum will not grow larger than the space available during the data processing steps, but no larger than necessary. For example, if you want to generate a 2048 by 4096 hypercomplex two-dimensional spectrum, you would invoke section and give respond with 2048 4096 4 to the prompt. The program multiplies the sizes of the dimensions together to determine the size of the shared memory section. Once it has created the section, it asks you if you want to remove it. If you are happy with the size, say no.

[31] section

Current NMR UID: hoch      2101 (0x0835)


Found 1 NMR shared memory sections

UID: fezoui    2105 (0x0839)   Attached: 2    Size: 131073 pages

Did not find your shared memory section: No such file or directory

Enter the dimensions (real points): 2048 4096 4

Created your NMR shared memory section.

Section size = 262145 pages.


Attached at address 600000.

Detached from the shared memory section.

Remove the shared memory section? n

All done.


Section is also used to remove existing shared memory sections. If you already have an active section, it will simply ask you if you want to remove it.


[32] section

Current NMR UID: hoch      2101 (0x0835)


Found 2 NMR shared memory sections

UID: hoch      2101 (0x0835)   Attached: 0    Size: 262145 pages

UID: fezoui    2105 (0x0839)   Attached: 2    Size: 131073 pages

Found your NMR shared memory section.

Section size = 262145 pages.


Attached at address 600000.

Detached from the shared memory section.

Remove the shared memory section? y

Removal was successful.

All done.


  1. 3.Invoke RNMRTK

Start RNMRTK by typing its name, as lowercase; it responds with the prompt

[33] rnmrtk

RNMRTK>


If the shared memory section doesn't exist, it complains and quits:

[34] rnmrtk

Could not find shared memory section.


If you see this message you need to go back to Step 2.

  1. 4.-- Load the data, process it, and plot it!

  2. 5.-- Don't forget to remove the shared memory section when you are done!