Antlr4 Eclipse and Gradle

November 12th, 2017

For my DSL ‘s the preferred tool is ANLTR by Terence Parr and his team (although I sometimes like to hone my Flex/Lex, Bison/Yacc skills too). With the new ANLTR4 creating your own compiler front end has never been so easy.

It still took me a little bit of time to combine it with Eclipse and Gradle. I did the following to make it work:

  • Install Antlr4 by Edgar Espina from eclipse market place
  • Create ANLTR project New->General->ANLTR 4 Project
  • Add grammar file e.g. src/main/antlr/<package>/mygrammar.g4
  • Important. Add header to your grammar with package . This will make sure that the package structure is created in the jar file like in the following example:

 

grammar TimeParser;

@header {
    package eu.informationsuperhighway.timeparser;
    import java.util.HashMap;
}
  • Right Click the project->Build Path->Source Path->Click ‘Add Source Path’->Click ‘Browse’
  • Browse to build->generated-src->antlr->main and make sure the checkbox for main is checked. This is where the generated sources end up and if it is not listed as the source dir then the calling project won’t be able to find the sources! You might need to compile first before this directory shows up.
  • You can right click on the <grammar>.g4 file to set options (although I had to restart eclipse to make them take effect). I you do a Gradle build the following Gradle file should do. You might need to upgrade Gradle so it recognizes the antlr4 plugin (execute task wrapper).
  • Add Gradle Nature to the project by right clicking on the project Configure->Add Gradle Nature
  • Refresh Gradle in the ‘Gradle Tasks’ view with the ‘Refresh All Projects’ button.
  • Double click the ‘Jar’ taks in Gradle tasks.
  • I used the following Gradle file to create a library to include into other projects. I am only interested in the ‘Visitor Design Pattern’ so I added ‘-no-listener’ and ‘-visitor’  for the arguments parameter ( see end of this post).
  • Note that if this is a library (probably), in the calling project, under dependencies add ‘compile project( ‘:<your project name>’ )’  and in settings.gradle add:

 

include ':<your project name>'
project(':<your project name>').projectDir = new File(settingsDir, '../<your project name>')

e.g.

include ':timeparser'
project(':timeparser').projectDir = new File(settingsDir, '../timeparser')
  • Also in the calling project’s buildpath add this project as dependent project (so the jar gets included) Right Click project->Build Path->Configure Build Path->Projects-> Add the library project
  • For verification; Note that the library’s project’s jar should be in workspace\<project>\build\libs.
/*
 * This build file was generated by the Gradle 'init' task.
 *
 * This generated file contains a commented-out sample Java project to get you started.
 * For more details take a look at the Java Quickstart chapter in the Gradle
 * user guide available at https://docs.gradle.org/3.5/userguide/tutorial_java_projects.html
 */


// Apply the java plugin to add support for Java
apply plugin: 'antlr'
apply plugin: 'java-library'
// In this section you declare where to find the dependencies of your project
repositories 
{
    // Use 'jcenter' for resolving your dependencies.
    // You can declare any Maven/Ivy/file repository here.
    jcenter()
    mavenCentral()
}

task wrapper(type: Wrapper) {
    gradleVersion = '4.3.1'
}

task compile(type: JavaCompile)

generateGrammarSource 
{
    maxHeapSize = "64m"
    arguments += ["-visitor", "-long-messages", "-no-listener"]
}

// In this section you declare the dependencies for your production and test code
dependencies 
{
    // The production code uses the SLF4J logging API at compile time
    //compile 'org.slf4j:slf4j-api:1.7.22'

    // Declare the dependency for your favourite test framework you want to use in your tests.
    // TestNG is also supported by the Gradle Test task. Just change the
    // testCompile dependency to testCompile 'org.testng:testng:6.8.1' and add
    // 'test.useTestNG()' to your build script.
    testCompile 'junit:junit:4.12'
    
     // https://mvnrepository.com/artifact/org.antlr/antlr4-runtime
    compile group: 'org.antlr', name: 'antlr4-runtime', version: '4.7'
	
    antlr "org.antlr:antlr4:4.7" // use ANTLR version 4 
}
 
jar 
{
    archiveName = "timeparser.jar"
    from 
    {

        configurations.runtime.collect { it.isDirectory() ? it : zipTree(it) }

        configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
    }
    
    from( "timeparser/build/generated-src/antlr/main" )
    

    manifest 
    {
       //attributes 'Main-Class': 'eu.informationsuperhighway.something.Main'
    }

    exclude 'META-INF/*.RSA', 'META-INF/*.SF','META-INF/*.DSA'    
}

Working capital

September 24th, 2017

Trying to understand accounting (better). I got pretty confused about working capital in combination with cash flow. Most sites seem to have the minus/plus signs wrong even in the book I am reading. So here goes my stab at it.

First off earnings, Net Income, Accounts payable etc in this article all actually refer to increases/decreases not absolute values.

Just for the record if I say ‘increase in X’ it means that if I first had 5 and later 10, the increase is a positive number 5. If I say ‘decrease in X’ then if I first had 10 and then 5 the decrease is a positive number 5.

Short story

Net Income is the money we received during this period and you could say we could add this to what we have in cash (i.e. real tangible money (well, OK, money in our bank account, counts too)).  However money we have yet to receive from our customers is also in this number since it are earnings but we haven’t received the money yet. The reverse for bills we have to pay, although not included in our Net Income (since theoretically we don’t have it anymore) we still have it in cash since we didn’t pay those bills yet.

Also if we increase our inventory we would have payed that with cash. This is not visible in the income statement since it doesn’t matter for our net income if we used that money to buy more inventory. But obviously we would have less cash now. So an increase in inventory will decrease our cash.

So instead of just adding the Net Income and therefore increasing what we have in cash we have to subtract the increase in accounts receivable, subtract increase in inventory, while adding the increase in accounts payable.

Long Story

Working Capital (Capital tied up in the company) = Current Assets (e.g. Inventory, Accounts Receivable) – Current liabilities (e.g. Accounts Payable)
Inventory (using LIFO or FIFO (goods in inventory are valuated against the old price, reasoning is that the new coming in are already sold so only the old goods are in the inventory, vice versa for LIFO).
On the cash flow statement we subtract changes in working capital, from the Net Income, since the more we have in working capital the less we can see in cash from what we earned.

Net income includes increase in Accounts Receivable, Minus increase in accounts payable. For Net Income it doesn’t matter we bought more inventory also though Accounts Receivable for instance increased this is not money we actually have in cash. So for the cash flow statement we have to subtract the increase in accounts receivable and subtract increase in inventory to understand how much our cash increased while adding the increase in accounts payable since an increase in Accounts Payable means we didn’t pay those bills and we still have it in cash.

Income statement (simplified)
Revenue
-COGS (Cost of goods sold, or cost of revenue)
+ Increase in Accounts Receivable
– Increase in Accounts Payable
Other stuff  (for instance amortization and depreciation (subtract those too))
+—————
Net Income

Cash flow (simplified)
Net Income What we have in cash would increase by the Net Income, but
– Increase in accounts receivable because we don’t have this in cash yet
+ Increase in accounts payable because we have it in cash because we didn’t pay those bills yet.

–  Increase in inventory since we used some of our cash for new inventory

Other stuff (for instance amortization and depreciation (add those too))
+—————
Increase in cash
Cash previous period
+—————
Total in cash now

 

Or using increase in working capital

Increase in Cash = Net Income – Increase in working capital + Other stuff

 

Adding the Android Licensing Library to an AndroidStudio Project

May 13th, 2016

It appears that currently creating an external library is not all that easy. I wanted to add the Android Licensing Library and they advise creating a copy from it. I decided to just use a project library instead of messing around but have the sources in the repo. Here is what I did:

  • Download the licensing library with the SDK Manager
  • Copy it from the android SDK (the location is visible in AndroidStudio->Tools->Android->SDK Manager) to another location
  • Check in the library into your version control system
  • In your AndroidStudio project create a library project. Projects->New Module->Library ‘SomeName’ Package name = com.google.android.vendor.licensing
  • Now with your file system browser go into the ‘SomeName’->src->main->java directory and delete the package that was created
  • Check out the licensing library into a new empty directory somewhere and go into the src dir and copy the ‘com’ subfolder
  • Copy this tree to the  ‘SomeName’->src->main->java directory (this makes sure the .svn references are now pointing to the correct repo location)
  • Now in your main project open gradle.build and make sure the new module is listed there (I had to add it)
    dependencies
    {
        compile project(':assetLib')
        compile project(':openGLISH')
        compile project(':mathLib')
        compile project(':ishlicensing')
        compile 'com.android.support:support-v4:18.0.0'
    }
  • Now you should be able to compile your project and check everything into svn. (note uncheck ‘Code Analysis’ if pressing the ‘Commit’ button doesn’t do anything).

 

Measuring Distance with an HC-SR04

February 16th, 2016

The HC-SR04 is a cheap (in terms of money)  module that can measure distances using ultrasound. The module uses a 40 KHz tone, which is of course way above what any human can hear, to measure the distance. It works as follows.

  • It expects a 10µs TTL, i.e. 0-5V, pulse on its ‘Trigger’ input
  • It will then send out a sound
  • Waits until the sound comes back and calculate the distance
  • Create a pulse on the ‘Echo’ output. The length of the pulse will correspond to the distance measured

I used a ATMega328p MCU to control the HC-SR04.

HCSR04

The yellow channel shows the outgoing pulse to the ‘Trigger’ input of the HCSR04. The blue channel shows the signal coming back on the ‘Echo’ output the length of it being proportional to the distance measure. The OLED display shows the values measured. The first few values are for debugging the last shows the measured and calculated distance in meters. In the top right the HCSR04 is visible sitting on my laser distance meter. Compare the value of that 0.156 to the one on the OLED display 0.150. They are pretty close.

The main parts of the program consist of:

  • an ISR (Interrupt Service Routine) to be called when the echo pin goes to high
  • a timer to measure the length of the pulse sent by the HC-SR04

One thing to keep in mind that it is not the time between starting the HC-SR04 and the start of the echo pulse that should be measured but the length of the echo pulse. That is to say it is not really the echo the HC-SR04 just sends you back the measure distance and the simplest way is to vary the length of the signal.

My ATMega328p is clocked at 8MHz. The supply voltate is 3.3V to stay compatible with my RPi. The HCSR04 however needs to be connected to a 5V supply. Obviously you need to connect the grounds of both power supplies to make it work. The ‘Trigger’ input was connected directly to the ATMega328p since it produces a 3.3V high and that is well within the limits of what is considered ‘high’ in TTL. However the ‘Echo’ pin I connected through a two resistor voltage divider (i.e. two resistors in series, one connected to the ‘Echo’ ouput and the other to ground while the middle is the ‘Echo’ that goes to the ATMega. The one connected to the ground divided by both, times the 5V should be somewhat below 3.3V.

Note that the code should be optimized, obviously, for any real application. Note also that the following is not complete (since I already deleted the code but I’ll recreate it some time and update this) just showing the idea. I used the OLED from a previous post for the output.

volatile uint16_t MeasureCount;
ISR (INT0_vect)
{
	// Reset timer
	TCNT1H = 0;
	TCNT1L = 0;
	// Wait until pind goes back to zero
	
	while( PIND & EchoPin );
	
	uint16_t LSB = TCNT1L;
	uint16_t MSB = TCNT1H;
	uint16_t Pulses = ( MSB << 8 ) | LSB;
	//Pulses = 65535;
	float Distance = ( (float) Pulses * 343.2f ) / ( 2.0f * 8000000.0f ) + 0.0005f;
	uint16_t Whole = (int)Distance;
	uint16_t Partial = (int)( ( Distance  - Whole ) * 1000.0f );
	++MeasureCount;
	sprintf( Message0, "M %06u H = %u L = %u %u", MeasureCount, MSB, LSB, Pulses );
	sprintf( Message4, "%u.%03u Meters", Whole, Partial );
}

void Init()
{
	cli();
	strcpy( (char*)Message4, "Started Hello World");
	DDRB = 0;  // All pins input
	DDRB |= 1;
	DDRB  |= BlinkPin; // Blinkping output
	PORTB &= ~BlinkPin; // Make pins low to start

	// turns on pin change interrupts
	PCICR |= 1;
	//
	DDRB |= PulsePin; // Pulse pin output
	PORTB  |= EchoPin;    // Enable pull up
	PCMSK0 |= EchoPin;    // turn on interrupts on pins
	
	sei();
	
}
u8g_t u8g;

void u8g_setup(void)
{
	u8g_InitI2C(&u8g, &u8g_dev_ssd1306_128x64_i2c, U8G_I2C_OPT_NONE);
	u8g_Begin( &u8g );
}
void StartMeasure()
{
	cli();
	PORTB |= PulsePin;
	_delay_us( 11 );
	PORTB & = ~PulsePin;
        // I used a small delay here, sometimes it seemed to immediately start measuring, maybe it picked up the pulse I sent
        // So wait just a little bit
	_delay_us( 2 );
	sei();
}
void draw()
{
   u8g_SetFont(&u8g, u8g_font_6x10);

   u8g_DrawStr(&u8g, 0, 10, Message0 );
   u8g_DrawStr(&u8g, 0, 21, Message1 );
   u8g_DrawStr(&u8g, 0, 32, Message2 );
   u8g_DrawStr(&u8g, 0, 43, Message3 );
   u8g_DrawStr(&u8g, 0, 54, Message4 );
}

int main(void)
{
	Init();
	u8g_setup();
	
	strcpy( Message4, "Hello world" );
		
        StartMeasure();
	u8g_FirstPage(&u8g);
        while(1)
        {
	  do
	  {

	    draw();
	  } 
          while ( u8g_NextPage( &u8g ) );
     }
  }
	
}

 

Galaxy S3 Compass Incorrect Values

February 7th, 2016

Hmph, spent an afternoon debugging my Galaxy S3 Sensor app since it stopped working, checked all code changes for the last year then checked all the transformations etc. until I realized the compass sensor itself gave invalid values.

As a reminder to myself. Seems that you sometimes need to calibrate it:

  1. Phone
  2. *#0*#
  3. In the test menu press sensors
  4. Immediately start rotating the phone around all its axes until the little compass at the bottom of the screen shows 3.
  5. Then lay it flat on the table and check that it displays the correct values (comparing it with a real compass) while rotating it over 360°
  6. If not correct go back to the sensor menu and repeat from 3 onwards.

 

Controlling 1306 128×64 OLED Display with an ATMega328P

February 3rd, 2016

Note that I mostly build on what other people researched and just put those results together however it is nice to have everything in one place. All credits go to the various authors I link to except for the glue in between (i.e. Atmel Studio part). So this will be mainly a starting point to the various sites.

This is a lot of text and I probably will spice it up a bit in the future with pictures of scantily dressed women, I mean boring diagrams and photos of ATMega328p’s.

Ok, with that out of the way let’s start. First off what do we need:

  1. Raspberry Pi
  2. ATMega328P
  3. Breadboard
  4. Wires to connect stuff
  5. Optional 5 * 1K resistors (to protect your Raspberry Pi).
  6. And of course an OLED display
  7. Windows (I do it on windows you have to improvise for Linux however most or almost all tutorial are for Linux so you should be OK there).
  8. Atmel Studio

 

ATMega328P

ATMega328P’s are fond of breadboards and easily trapped in one. No worries though, I added a life support system to feed it.

 

Setting up and connecting the ATMega328P to your RPi

For that go to this site BUT instead of using the pin assignment for the ATTiny go to this site. Admire the ATMega328P with its many pins for a while and in the mean time I am going to get a cup of coffee while you  lookup and connect the corresponding pins for the ATMega328P.

So you’re finally back (I drank coffee, cleaned the house and destroyed some IC’s with static electricity in the mean time but that’s OK).

Atmel Studio

We will be using Atmel Studio to compile stuff and then scp the resulting binary to your Raspberry.

  1. Download Atmel Studio. You probably have to supply your email and you get some nice ‘Maker’ info from Atmel. Your are now an official ‘Maker’ 🙂
  2. Install Atmel Studio. This is Visual Studio under the hood so if you are familiar with it you feel right at home. If not and haven’t used any IDE before, you are probably looking at a confusing quagmire of User Interface Elements. Don’t worry though, I’ve got you covered, even if you haven’t got the remotest clue what you’re doing or even what we’re trying to accomplish here in the first place.
  3. Start it and after a while you arrive at the welcome page.
  4. Select ‘New Project…
  5. Select ‘GCC Executable Project’
  6. Give it an original name (like OLED).
  7. Now there is a device selection window (may not be the top window). Click on it and select  ATMega328P
  8. You are in your little project but as good cut & paste developers we are not going to program, yet…
  9. Instead, of writing the lower layers, I downloaded the UG8 library for AVR and build on the Herculean job other developers have done.
  10. Once downloaded unzip it and descent into the ug8lib/src dir.
  11. In another window go into your Atmel studio source dir (one deeper than where your *.atsln file is). So if you named the project OLED you should select that dir twice to get there.
  12. Select all ‘.c’ and ‘.h’ files and copy them to from ug8lib/src and copy them to the second window.
  13. Now in Atmel studio the right pane should contain ‘Solution Explorer’ (if you didn’t mess with the windows).
  14. Now right click on the project (not the solution which is the topmost node) and select ‘Add’->’Add Existing Item…’
  15. Select all ‘.c’ and ‘.h’ files that you copied there to add them to the project.
  16. Go to Build->Configuration Manager and select ‘Release’ as active build.

Almost there, our project is pretty much ready but we need to correct settings.

  1. Again in the ‘Solution Explorer’ (the right window). Right click again on the project node (still not the top node) and select properties.
  2. Basically if you look in the ug8lib dir there was a Makefile and I just set all the options according to that (if I could find them).
  3. Build->Configuration->Select Release
  4. Select ‘Toolchain’
  5. Select the following options for Avr Gnu Compiler:
  6. General: -mcall-prologues
  7. General: -funsigned-char
  8. General: -funsigned-bitfields
  9. Preprocessor: Unseselect everything (should already be unselected)
  10. Symbols: Click on the green + symbol and add ‘F_CPU=8000000’ this is your processor speed 8MHz.
  11. Optimization: Select Optimize for size (-Os) in the drop-down list
  12. Optimization: Select everything except -mshort-calls
  13. Warnings: Select -Wall
  14. AVR Gnu Linker:
  15. General: -WL  (actually not sure if it is an L or the number 1)
  16. Optimization: -WL –gc-sections

Then Build->Clean Build (not really necessary probably) followed by Build->Rebuild All. Now pray to your favorite God. I pray to the god of Jesus but you can pray to yours and we will see who’s build succeed :-).

If there are almost no errors. I got a warning about a pointer that was incompatible, however when I searched the source code for that, it was because they defined a function pointer as taking a parameter as ‘void *’ instead of the pointer to the function you assign it to, is using. Since they were the same size it isn’t a problem.

Now go into the OLED dir where you copied the .c and .h file and there should be a directory ‘Release’. If there isn’t but there is a ‘Debug’ instead you didn’t select the right project build in Build->Configuration Manager.

In that directory there should be a file ending in *.hex and that file is your program and need to be copied to your Raspberry Pi.

Modify the Raspberry Pi project for your hex file

Now in the source I assume the clock is 8MHz (#define F_CPU  8000000) so we must ‘fuse‘ the ATMega to 8MHz (i.e. turn of the clock divider):

sudo /usr/local/bin/avrdude -p m328p -P /dev/spidev0.0 -c linuxspi -b 10000 -U lfuse:w:0xE2:m

When you went through the instruction on this site, you ended up with a blinky program in the blinky directory.

cp -R that blinky dir to an OLED dir and instead of the Makefile we can use the following script (I just looked at the Makefile from this  site so all credits go to the original author ):

sudo gpio -g mode 22 out
sudo gpio -g write 22 0
sudo /usr/local/bin/avrdude -p m328p -P /dev/spidev0.0 -c linuxspi -b 10000 -U flash:w:OLED.hex
sudo gpio -g write 22 1

 

By running the above commands you flash your program to the ATMega328p.

It should write and verify just as with the blinky app.

 

Hooking up the Display

With the display in front of you the pin assignment is GND, VCC, CLCK, DATA. Make sure to flip it vertically so they stay in the same order now connect them as follows:

  • GND connect to the GND of you breadboard
  • Make sure you use the 3.3 V of your RPi not the 5V.
  • VCC connect to the 3.3V
  • CLK connect to the upper right pin (assuming pin 1 (with the notch) is at the top left) of the ATMega328p i.e. pin 28 top right pin
  • DATA connect to the second pin at the upper right (i.e. the pin just below where we connected CLK) i.e. pin 27

Grand Finale

Connect the reset pin i.e. the upper pin at the left (the one with the notch) to Gnd and then to the VCC to reset the device.

The device should now display ‘Hello world’. In the picture below it shows ‘Does it work?’ because I attached it to my RF433 receiver (and I sent that string from my RPi).

OLDE

Final result. The (pretty bad) signal is from my RF433Mhz receiver. My RPi sends a string using a RF433 transmitter which is then received and decoded by an assembly routine I wrote for the ATMega328.  The resulting string is then displayed instead of the standard ‘Hello world’ string. Admittedly currently the string is only correctly decoded once in several times due to noise on the wires.

 

 

 

 

wpa_supplicant Excessive Logging on Raspberry Pi

December 25th, 2015

Today I noticed there was excessive logging from wpa_supplicant on my RPI. Normally this is of course not a problem but if you run it on an SDC like I do it might wear the memory since there is a low limit on the number of times you can rewrite it (in comparison with a harddisk).

I think there is a parameter $IF_WPA_VERBOSITY that can be set somewhere (maybe in the network file) however I was not in the mood to spend time on it so I just changed the following code in /etc/wpa_supplicant/functions.sh


WPA_SUP_OPTIONS="-s -B -P $WPA_SUP_PIDFILE -i $WPA_IFACE"


WPA_SUP_OPTIONS="-qq -f /dev/null -s -B -P $WPA_SUP_PIDFILE -i $WPA_IFACE"

After that is seemed to have stopped. Need to do it again after an update/upgrade though.

Kung Fury

May 31st, 2015

What movie can pack dinosaurs, kung-fu, tricerocop’s, hitler (AKA the ‘kung führer’), thor, Vikings and time travel in only 30 Minutes? It’s got to be ‘Kung Fury’ one amazingly short movie. As Jon Irenicus would say, ‘how wonderfully mad’.

This movie is packed with action, fantasy, FX on every frame and is completely free (though you might want to buy the game or movie to support them).

If you like humor, action and FX you might want to watch .

Math MediaWiki extension with Texvc problem.

May 28th, 2015

This puzzled me for a while. My math was rendered as $ Hard to understand math goes here $ instead of a nice png. I don’t fully understand the problem but made it work anyway by hacking the Math extension sources.

First to make it work texvc must be installed (obviously). You must be able to run it as your web server (whatever that account is running on). So far the obvious part. Now you have to add the following to your LocalSettings.php in the root wiki dir in this order:

$wgMathValidModes[] = array( MW_MATH_MATHJAX, MW_MATH_PNG, MW_MATH_SOURCE ); // Define MathJax as one of the valid math rendering modes
$wgUseMathJax = false; // Enable MathJax as a math rendering option for users to pick
$wgDefaultUserOptions['math'] = MW_MATH_SOURCE; // Set MathJax as the default rendering option for all users (optional)
$wgMathDisableTexFilter = true; // or compile "texvccheck"
$wgUseTeX = true;
$wgTexvc = '/usr/bin/texvc';
$wgUploadDirectory= "upload";
$wgUploadPath = "upload";
$wgMathPath = "{$wgUploadPath}/math";
$wgMathDirectory = "{$wgUploadDirectory}/math";
$wgTmpDirectory = "{$wgUploadDirectory}/tmp";
$wgShowSQLErrors = 1;
require_once "$IP/extensions/Math/Math.php";

Make sure the directories are created under your root wiki install so if the LocalSettings.php are in /home/wiki/ then there should be a dir

  • /home/wiki/upload
  • /home/wiki/upload/math and
  • /home/wiki/math/tmp

(you can of course use other locations to make it more difficult to get it working 🙂 ).

Now for some reason the tmpDir ad wgTexvc don’t make it into the following file (don’t ask me why) or maybe tmpDir is not defined altogether. So I just hardcoded them (don’t forget to redo it if it works and you do an upgrade some time in the future). Update extensions/Math/MathTexvc.php as follows:

public function callTexvc() {
global $wgTexvc, $wgTexvcBackgroundColor, $wgHooks;
$wgTexvc="/usr/bin/texvc";
wfProfileIn( __METHOD__ );
$tmpDir = wfTempDir();
if ( !is_executable( $wgTexvc ) ) {
wfDebugLog( 'texvc', "$wgTexvc does not exist or is not executable." );
wfProfileOut( __METHOD__ );
return $this->getError( 'math_notexvc' );
}
$tmpDir="/tmp";

 

Then finally they appeared in the preview. But unfortunately my wiki is not in the root of the website and it didn’t use the complete URL in the final pages as it does in the preview. This can be solved as follows in MathTexvc.php. I just hardcoded (again) the prefix.

return Xml::element( 'img',
$this->getAttributes(
'img',
$attributes,
array(
'src' => "/mydir/mywiki/".$url

After that my png’s appear nicely in my wiki, great.

 

Hopes this helps someone out there.

Shrink partition in xen guest

April 25th, 2015

In a previous post I discussed extending a partition in a linux guest. Since I never can make up my mind, this post discusses the opposite, i.e. shrinking a paritition in a disk (again as opposed to a partition) in a logical volume. Since we do not have just one partition in the logical volume we have to delete and recreate.

The procedure is a little different since it is too large to make a backup in the other quest I do it on the xen host:

  • Create a logical volume the size of the one we are going to rearrange for backup
  • Mount the disk in the Logical Volume (LV) on another guest.
  • In the other guest use fdisk to recreate them with the right size

 

lvcreate -L 100G -n lv_vm_tmp /dev//

For instance:

lvcreate -L 100G -n lv_vm_ishpublic_tmp /dev/nebulus-vg

Backup the original volume to the temp volume we just created (double check that the original is first and the backup is second).

cp /dev// /dev//
cp /dev/nebulus-vg/lv_vm_ishpublic /dev/nebulus-vg/lv_vm_ishpublic_tmp

Now add this to your ‘utility’ host (we are going to mount the volume there to resize). In your xen config for utility add the disk. For instance my volume group is ‘nebulus-vg’ and the disk with partitions I want to resize is ‘lv_vm_ishpublic’.

disk = [ '/dev/nebulus-vg/lv_sirius_ubuntu,raw,xvda,rw', '/dev/nebulus-vg/lv_vm_ishpublic,raw,xvdb,rw' ]

Then:

  • Start your utility VM
  • login as root

Print the partitions on xvdb

root@sirius:~# fdisk /dev/xvdb

Welcome to fdisk (util-linux 2.25.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/xvdb: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x76ff0726

Device     Boot     Start       End   Sectors Size Id Type
/dev/xvdb1           2048 201328639 201326592  96G 83 Linux
/dev/xvdb2      201330686 209713151   8382466   4G  5 Extended
/dev/xvdb5      201330688 209713151   8382464   4G 82 Linux swap / Solaris

Partition 3 does not start on physical sector boundary.


Command (m for help): q

Run a file system check:

e2fsck -f /dev/xvdb1

Try resizing the partition (we can pass a very low number to resize2fs so it will print the minimum we need:

resize2fs /dev/xvdb1 1G
resize2fs 1.42.10 (18-May-2014)
resize2fs: New size smaller than minimum (4468729)

You probably do not want to have the exact minimum, I use approx. double the size.

resize2fs /dev/xvdb1 8388608
resize2fs 1.42.10 (18-May-2014)
Resizing the filesystem on /dev/xvdb1 to 8388608 (4k) blocks.
The filesystem on /dev/xvdb1 is now 8388608 blocks long.

Though the filesystem is resized the partition is still the original size. We need to modify the partition table on the disk where the new end is. Calculate this from the output of resize2fs and the size of a sector from fdisk (sector size was 512 in my case, make sure you substitute the values you got!).

  • New filesystem size (output from resize2fs) = 8388608 (4k) blocks = 8388608 * 4 * 1024 = 34359738368 bytes
  • New filesystem size in sectors = 34359738368 / 512 = 67108864 sectors (if you get a fractional value round it up)
  • Therefore the end sector in fdisk will be start sector + size = 2048 + 67108864 = 67110912
  • I also want a 2G swap partition i.e. 2 * 1024 * 1024 * 1024 / 512 = 4194304 sectors
  • Therefore the end of the disk will be at 67110912 + 4096 = 71305217 sectors
 fdisk /dev/xvdb

Welcome to fdisk (util-linux 2.25.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.


Command (m for help): p
Disk /dev/xvdb: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x76ff0726

Device     Boot     Start       End   Sectors Size Id Type
/dev/xvdb1           2048 201328639 201326592  96G 83 Linux
/dev/xvdb2      201330686 209713151   8382466   4G  5 Extended
/dev/xvdb5      201330688 209713151   8382464   4G 82 Linux swap / Solaris

Partition 3 does not start on physical sector boundary.


Command (m for help): d
Partition number (1,2,5, default 5):

Partition 5 has been deleted.

Command (m for help): d
Partition number (1,2, default 2):

Partition 2 has been deleted.

Command (m for help): d
Selected partition 1
Partition 1 has been deleted.

Create the new partitions:


 

Command (m for help): n Partition type p primary (0 primary, 0 extended, 4 free) e extended (container for logical partitions) Select (default p): Using default response p. Partition number (1-4, default 1): First sector (2048-209715199, default 2048): Last sector, +sectors or +size{K,M,G,T,P} (2048-209715199, default 209715199): 67110912 Created a new partition 1 of type 'Linux' and of size 32 GiB. Command (m for help): n Partition type p primary (1 primary, 0 extended, 3 free) e extended (container for logical partitions) Select (default p): Using default response p. Partition number (2-4, default 2): First sector (67110913-209715199, default 67112960): 67110913 Last sector, +sectors or +size{K,M,G,T,P} (67110913-209715199, default 209715199): 71305217 Created a new partition 2 of type 'Linux' and of size 2 GiB.

Toggle the filesystem type on the swap partition:

t 2 82

Write the table back to disk, quit and shutdown the utility system:


 

w q shutdown -h now

Back on the xen host we now need to reduce the logical volume size. First we need to know the physical extent:

 vgdisplay
  --- Volume group ---
  VG Name               nebulus-vg
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  12
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                7
  Open LV               5
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               3.64 TiB
  PE Size               4.00 MiB
  Total PE              953672
  Alloc PE / Size       129825 / 507.13 GiB
  Free  PE / Size       823847 / 3.14 TiB
  VG UUID               tm4Q7z-LWIL-OUW3-egF0-8UP4-d2MJ-fW1sXX


PE Size is 4 MB

To calculate the new size in extends take the size in bytes from our last sector ( I am not sure if sectors start from 1 or 0, better be safe) from fdisk and add one.

  • 71305217 + 1 = 71305218

 

Round up to the next vg extent boundary:

  • RoundUp( Number of sectors * sectors size / PE Size )
  • 71305218 * 512 / ( 4 * 1024 * 1024 ) = 8704.25
  • RoundUp( 8704.25 ) = 8705

Shrink the logical volume

lvreduce -l 8705 /dev/nebulus-vg/lv_vm_ishpublic
  WARNING: Reducing active logical volume to 34.00 GiB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce lv_vm_ishpublic? [y/n]: y
  Reducing logical volume lv_vm_ishpublic to 34.00 GiB
  Logical volume lv_vm_ishpublic successfully resized

And we are done. Well actually you would want to startup your utility VM again, do the mount of the filesystem and run e2fsck (see above) again to make sure that everything worked.

e2fsck /dev/xvdb1
e2fsck 1.42.10 (18-May-2014)
/dev/xvdb1: recovering journal
Setting free inodes count to 1775009 (was 1775029)
Setting free blocks count to 4183513 (was 4183757)
/dev/xvdb1: clean, 322143/2097152 files, 4205095/8388608 blocks

Bonus points enable swap.

Shutdown util and startup your VM with resized disk and login:

If you configured a swap parition check it is working:

swapon -s

If not you have to add the swap partition. Let’s say it was on xvdb2 when you configured it in util then it probably is in xvda2 (make sure you get it right i.e. check if it exists in /dev isn’t mounted df -k). In the following I assume it is on /dev/xvda2

Now for ubuntu execute the following:

blkid /dev/xvda2

Copy the UUID and open /etc/fstab. Copy the swap line and change it as follows (use the printed UUID instead of the one below).

UUID=7340ea5f-3325-466a-8e2f-52c1cf15ae94 / ext4 errors=remount-ro 0 1
# swap was on /dev/xvda5 during installation
#UUID=75ecaea3-ff0d-4f90-b5e2-ea757bde5399 none swap sw 0 0
UUID=582dae7a-5d2a-4b4c-9685-993fc8454200 /dev/xvda2 swap sw 0 0

shutdown -r now

Log back in:

swapon -s
Filename                                Type            Size    Used    Priority
/dev/xvda2                              partition       1046524 0       -1

If satisfied don’t forget to start Util and cleanup the backups created.