(I do not recommend watching this demo on a smartphone. It’s quite flashy and exacerbated my headache. Also, the WebMSX code will ask you to go fullscreen, but I don’t think you can start the demo without going fullscreen and then back again.)
In part 1, we constructed a 48 KB ROM to play a tune called “Popsa 2”. We didn’t apply any real compression algorithms but implemented a set of scripts to find repeated sections in a binary file and added an instruction to the .psg file format to “call” repeated sections. Using real compression algorithms we could achieve much better compression, and each tune would just occupy a couple KBs. Using our method, we _just_ manage to fit the tune into a single cartridge. Popsa 2 fit into 48 KB, and the tune we’re going to do today is going to require a 64 KB ROM. If the previous track didn’t quite do it for you, I think it might be worth giving this one a chance. It’s a very complex piece of wonder-inducing music in my opinion. (Press the power button and in the menu that pops up, choose “Power” to boot the ROM.)
64 KB ROMs still require a header at 0x4000 or 0x8000. This means that we need to add a header and some entrypoint code to set up the slots right in the middle of our data. That’s inconvenient, but I didn’t feel like changing the structure of the program, so I just added one check each at the beginning and end of the main loop to see if the HL register has gone above a certain value. If yes: before the main loop, it adds an offset; after the main loop, it subtracts the same offset again. This way, we don’t have to do anything too complex when jumping to a previous section of the track.
In part 1, I mentioned a problem in WebMSX that prevented the 48 KB ROM from working. 64 KB ROMs are not affected by this problem. The WebMSX player at the top of this article page plays the ROM linked to above. The ROM also works on real hardware (the Hitachi MB-H2 MSX1 I repaired a while ago).
Aside: disabling WebMSX’ auto-scroll
In the unlikely event that you have read this blog’s front page sometime in the last few months, you might have noticed that it scrolled automatically to this WebMSX player, even though this post is now very much not the newest post on this blog! I only noticed this a short while ago and decided to fix it, because it’s quite annoying. The below code snippets are taken from the WebMSX commit with the tag “v6.0.4”. Older or newer versions may look different.
All you need to do is remove the “this.focus()” line in the powerOn function in CanvasDisplay.js:
this.powerOn = function() {
this.setDefaults();
updateLogo();
document.documentElement.classList.add("wmsx-started");
setPageVisibilityHandling();
this.focus(); // <-- this is the line you need to remove or comment out
if (WMSXFullScreenSetup.shouldStartInFullScreen()) {
setFullscreenState(true);
if (FULLSCREEN_MODE !== 2 & isMobileDevice) setEnterFullscreenByAPIOnFirstTouch(); // Not if mode = 2 (Windowed)
}
};
If you prefer to just edit the minified version, search for the call to setPageVisibilityHandling() and then edit out the “this.focus(),” bit.
This article is somewhat technical. If you just want to listen to a chip tune on WebMSX, maybe go for part 2 instead.
In previous articles I explored the YM2151 and the VGM file format. In this article, we’ll go back a generation and listen to some tunes written for the DSG (doorbell sound generator) PSG (programmable sound generator, i.e., the General Instruments AY-3-8910, or compatibly, Yamaha’s YM2149). PSG files (and particularly ASC files) are mainly used for ZX Spectrum chip tunes (I think), but the MSX has the same sound chip so why not play some chip tunes on the MSX?
Well, before we spend time working on something just slightly above PC beeper music… are there even any decent PSG tunes? Well, I’ve found at least one that like, “Popsa 2”, as is included in the below mix (scroll down a bit) on YouTube for example, and some of the commenters on this video seem to like “Illusion”.
Unfortunately, it doesn’t work in WebMSX (after 20 seconds or so). But it works in all three (NTSC) openMSX machines I bothered to test with, and it also works on my real MSX1 (Hitachi MB-H2). To get it to run in WebMSX, you have to “Set ROM format” -> “KonamiSCC”, but even then it’ll crash after a few minutes (vs. 20 seconds for e.g. ASCII8). For some reason it doesn’t let me choose “Normal”. I’m quite sure it would work with that setting if it were available. :p I’ll look into the matter at some point, probably. Looks like WebMSX will require a patch to work. Patch is submitted and will probably make it into the next version.
This machine produces NTSC color artifacts like there is no tomorrow.
Caution: writing to certain PSG registers is unsafe on certain MSX machines. I don’t think my code writes to these registers, but I didn’t make 100% sure. (However, openMSX gives you a warning when it notices unsafe writes, and I didn’t get a warning.)
“Popsa 2” was made in a program called ASC Sound Master. The “.asc” file can be downloaded here: https://zxart.ee/eng/authors/d/dreamer/popsa-2/. These .asc files are pretty small. They can be converted to PSG using ZXTune (https://bitbucket.org/zxtune/zxtune/) (and from PSG they can easily be converted to e.g. VGM, see bottom of this post), but the resulting files are too large to fit on a regular MSX1 cartridge.
ZXTune compilation and conversion:
git clone https://bitbucket.org/zxtune/zxtune.git
cd zxtune
make platform=linux system.zlib=1 -C apps/zxtune123/ -j4
bin/linux/release/zxtune123 --convert mode=psg,filename=foo.psg -- Dreamer\ -\ POPSA-2\ \(1994\).asc
The original .asc file is 3720 bytes. The resulting .psg is 129028 bytes. If you convert that to VGM, the resulting size is 187342 bytes.
The PSG file format
The PSG file format is very similar in concept to the VGM file format, except that only one chip is supported, the PSG. It seems it’s primarily used for ZX Spectrum chip tunes. As only one chip is supported, you don’t need the “command byte” that indicates what chip is to be written to. So you only have pairs of “register address” and “register value to write”.
There’s also a header in the first 16 bytes. The first three bytes are “PSG”, dunno about the rest.
The PSG only has 16 (IIRC) registers, and some of those aren’t even relevant for sound. In other words, the registers 0x10 to 0xff don’t exist and the designers of this file format used that opportunity to fit in a “wait” command at 0xff (one raster scan, so 1/50s or 1/60s depending on whether the system is PAL or NTSC). There’s also a command that waits multiple raster intervals, 0xfe, and a command that ends the tune, 0xfd. Ignoring the header, here are the first few bytes of the Popsa 2 PSG file:
All this means: wait 1 raster interval, then write to registers 00 through 0a with values 41, 05, 0b, 01, …, respectively, wait 1 raster interval, write to registers 02, 03, 07, 09, 0a, with values e0, 00, 38, 0d, 0c, respectively, wait 1 raster interval. (As you can see the 0xff command doesn’t take any parameters.)
Now that we know mostly how this file format works, it’s time to think about how to fit roughly 126 KB of data into my 48 KB cartridge. We could easily use an off-the-shelf compression library, but where’s the fun in that? That’s like… modern programming, ew.
We’ll invent another command for PSG, 0xfc, which takes a two-byte parameter that tells it to jump back somewhere (for a while, and then returns to its original location). We also need to write a program that identifies repetitive sections in the music (of which there are plenty). The former is pretty easy, so let’s talk about the latter program first.
Compute MD5 sums of a 100-byte window for every byte in the file. So we end up with 129028-100=128928 MD5 sums. Easy and fast on modern hardware. See code snippet below.
Check if we even have repeated chunks, e.g. by executing: md5sum chunks/* | awk ‘{print $1}’ | sort -n | uniq -c
We may want to check a couple other window sizes to see if we can get better results. A lower window size means we’ll find more repetition, but we need 3 bytes to encode a jump in our PSG file.
Re-assemble PSG file using a quick-and-dirty and probably somewhat buggy script. (See below.)
The resulting data length is 42158 bytes for the Popsa 2 song.
For task (1) we first convert the PSG file into hex, and later into tokens:
xxd -p Dreamer\ -\ POPSA-2\ \(1994\).psg | sed -r -e 's/(..)/\1 /g' | tr -d '\n' > Dreamer\ -\ POPSA-2\ \(1994\).psg.hex
# Then remove 16-byte header using a standard text editor
Then divide the tokens into chunks using the below script, divide_tokens_into_chunks.sh:
#!/bin/bash
N=100 # sliding window length
mkdir -p chunks_N$N
line_count=$(cat tokens | wc -l)
for ((i=0; i<$((line_count-N)); i++)); do
tail -n +$i tokens | head -n $N > chunks_N$N/chunk_$i
done
rm chunks_N$N/chunk_0 # same as chunk_1
You know, looking back at this code for the first time in a while, I see there’s a nice off-by-1 error and a nice rm command to fix half of the problem. But the great thing about this being a hobby is that I don’t need to care. :)
Next, we have a Perl script that creates our PSG file. It needs some help though, so we do this first:
(We can’t do md5sum chunks_N100/* because that expands to a tad too many arguments in our case. xargs automatically cuts down the number of arguments to a more reasonable value.) This is the main program. Usage: ./compress_aggressive_but_convert_to_psg.pl < chunks_N100_md5sums > foo.psg
#!/usr/bin/perl
# dependencies:
# chunks_N$N/ (directory)
# chunks_N$10_md5sums (file) # example generation: find chunks_N10/ | xargs md5sum > chunks_N10_md5sums
use strict;
use warnings;
use feature "switch";
my $N = 100;
my $md5s = {};
my @chunks;
my $md5;
my $file;
my $debug_logged = 0;
my $lines = [];
my $current_output_byte_number = 0;
for (my $chunk_number = 0; <>; $chunk_number++) {
/([a-z0-9]+)\s+([a-zA-Z0-9_\/]+)/;
$md5 = $1;
$file = $2;
if (exists $md5s->{$md5}) {
# can't call chunks that already contain a call because that call would take us beyond the N token window that we can see from where we are
# that means it's likely we'd generate wrong code
# so we'll just move on and maybe we'll find a nicer block
my $target_chunk_number = $md5s->{$md5}->{chunk_number};
my $concatted_chunks = join('', @chunks[max(0, $target_chunk_number-$N)..min($#chunks, $target_chunk_number+$N)]);
if (($concatted_chunks =~ /; call/) or # NOTE "call wait_for_raster" is allowed
($chunk_number - $target_chunk_number < $N)) {
# 1) can't convert due to existing call; nothing to be done here, or
# 2) we can't call something right behind us
# DANGER let's head back to the non-exists path
goto NON_EXIST_PATH;
} else {
if (!$md5s->{$md5}->{converted_to_call}) {
convert_to_callable_sub($target_chunk_number);
$md5s->{$md5}->{converted_to_call} = 1;
}
my $output_byte_number_high = int($md5s->{$md5}->{output_byte_number} / 256);
my $output_byte_number_low = $md5s->{$md5}->{output_byte_number} % 256;
$chunks[$chunk_number] = sprintf("fc %02x %02x ; call " . $md5s->{$md5}->{output_byte_number} . " ($md5)\n", $output_byte_number_high, $output_byte_number_low);
$current_output_byte_number += 3;
# skip next N-1 rows
for (0..$N-1) {
my $foo = <>;
$chunk_number++;
$chunks[$chunk_number] = "";
}
}
} else {
$md5s->{$md5} = {};
$md5s->{$md5}->{chunk_number} = $chunk_number;
$md5s->{$md5}->{converted_to_call} = 0;
NON_EXIST_PATH:
open my $fh, '<', $file or die "Can't open \"$file\": $!";
my $token = <$fh>;
close $fh;
my $asm = convert_to_asm($token);
$md5s->{$md5}->{output_byte_number} = $current_output_byte_number;
$current_output_byte_number += (scalar(split(" ", $asm)));
$chunks[$chunk_number] = $asm;
}
}
print foreach @chunks;
print "infloop:
jr infloop\n";
# no changes needed
sub convert_to_callable_sub($) {
my $block_number = shift;
}
# don't actually do anything here
sub convert_to_asm($) {
my $string = shift;
return "$string";
}
sub min($$) {
my ($a, $b) = @_;
return $a if ($a < $b);
return $b;
}
sub max($$) {
my ($a, $b) = @_;
return $a if ($a > $b);
return $b;
}
The output of this program is in hex. Now we just need some assembly code to read the data and put it into the PSG registers. Here’s the core part:
ld hl,psg_begin
main_loop:
ld a,(hl)
cp 0xff
jr z,wait
cp 0xfe
jr z,wait_n_times
cp 0xfd
jr z,end
cp 0xfc
jr z,jump
jr register_write
inc_loop:
inc hl
jr loop
wait:
call wait_for_raster
jr inc_loop
register_write:
ld a,(hl)
out (0xa0),a
inc hl
ld a,(hl)
out (0xa1),a
jr inc_loop
wait_for_raster:
in a,(0x99)
and 128
cp 128
jr nz,wait_for_raster
ret
psg_begin:
include "foo.psg"
ds 010000h-$ ; fill rest with 0s
Understanding the above should help understanding the full implementation. (The above doesn’t include the code for the 0xfe, 0xfd, and 0xfc commands.) Note that we can’t use the above wait_for_raster on NTSC machines because the tune assumes 50 Hz. So we’ll instead emulate the 50 Hz interval using a busy loop.
For 0xfd (end of song), we just enter an infinite loop. For 0xfe, we just call wait_for_raster multiple times. For 0xfc, we need to store where we left off, then set hl to the address in the parameter, then execute exactly 100 main loop runs, then set hl back to its previous address and continue as normal.
Here’s the code, which also includes some VRAM writes to visualize the music a little bit. Does it look good? Eh, I dunno. It was an experiment. I changed the registers to be displayed because some registers don’t see updates very often. The overall visuals are a bit noisy, but there is one section that looks good in my opinion, and it’s also the section that I like best in the tune, right at the end. You can clearly see one of the registers changing right in sync with the doorbell sound. (It looks even more in sync in openMSX.)
N: equ 100
org 4000H
db "AB"
dw entry_point
db 00,00,00,00,00,00,00,00,00,00,00,00
SetVdpWrite: macro high low ; from http://map.grauw.nl/articles/vdp_tut.php
ld a,low
out (0x99),a
ld a,high
add 0x40
out (0x99),a
endm
vpoke: macro value
ld a,value
out (0x98),a
endm
entry_point:
; copy cart rom (c000-f000) to ram
in a,(0a8h)
and 11000000b ; we want to know which slot is RAM, and AFAIK RAM should be mapped in at 0xc000-0xffff.
ld c,a ; save value for later
in a,(0a8h)
and 00001100b ; we are executing from cartridge ROM at 0x4000~0x7fff, so the 2-bit value for this region is known correct. we just have to make the slots above this one the same value.
ld b,a ; save a
rla ; << 1 (now have 000xx000b)
rla ; << 1 (now have 00xx0000b)
or b ; | saved b (now have 00xxxx00b)
rla ; << 1 (now have 0xxxx000b)
rla ; << 1 (now have xxxx0000b)
or b ; | saved b (now have xxxxxx00b)
; ld a,01010100b ; set pages 0: rom 1: rom 2: cart 3: cart
out (0a8h),a
copy_c000_f000:
ld hl,0c000h ; start at c000
copy_c000_f000_loop:
ld a,(hl) ; read from ROM address (hl)
ld d,a
in a,(0a8h)
ld b,a ; store original value
and 00111111b ; only keep settings for lower three slots
or c ; add in setting for top slot (saved earlier)
; ld a,011010100b
out (0a8h),a ; set port
ld (hl),d ; store value read from ROM address (hl) to RAM address (also hl of course)
ld a,b ; load a with original value
out (0a8h),a ; set port back
inc hl
ld a,h
cp 0f0h
jp z,other_init ; done with this copy
jp copy_c000_f000_loop
; entry_point:
; ld a,0xd4
; out (0xa8),a ; set slots
other_init:
; set ports to bios:cart:cart:ram
in a,(0a8h)
and 00111111b ; only keep settings for lower three slots
or c ; add in setting for top slot (saved earlier)
out (0a8h),a ; set port
; set colors
ld a,011110000b ; set data to be written into register (white on black)
out (099h),a
ld a,010000111b ; set register number (7)
out (099h),a
SetVdpWrite 0x20 0x05
vpoke 0x0f ; set white on black for some part of the screen
vpoke 0x0f ; set white on black for some other part of the screen
video_init:
; put chars /0123456789 into 0x1800-0x1AFF
SetVdpWrite 0x18 0x00
ld b,64 ; 64 chars
video_loop_1:
vpoke 0x2f
djnz video_loop_1
ld b,64 ; 64 chars
video_loop_2:
vpoke 0x2f
djnz video_loop_2
ld b,64 ; 64 chars
video_loop_3:
vpoke 0x33
djnz video_loop_3
ld b,64 ; 64 chars
video_loop_4:
vpoke 0x33
djnz video_loop_4
ld b,64 ; 64 chars
video_loop_5:
vpoke 0x36
djnz video_loop_5
ld b,64 ; 64 chars
video_loop_6:
vpoke 0x36
djnz video_loop_6
ld b,64 ; 64 chars
video_loop_7:
vpoke 0x31
djnz video_loop_7
ld b,64 ; 64 chars
video_loop_8:
vpoke 0x31
djnz video_loop_8
ld b,64 ; 64 chars
video_loop_9:
vpoke 0x35
djnz video_loop_9
ld b,64 ; 64 chars
video_loop_10:
vpoke 0x35
djnz video_loop_10
ld b,64 ; 64 chars
video_loop_11:
vpoke 0x37
djnz video_loop_11
ld b,64 ; 64 chars
video_loop_12:
vpoke 0x37
djnz video_loop_12
ld b,64 ; 64 chars
ld b,0 ; flag to indicate whether we are jumping around at the moment (0 means we aren't) (NOTE: nested jumping isn't supported)
ld c,0xa0 ; first PSG port
ld hl,psg_begin
jr main_loop
loop:
ld a,b
cp 0
jr z,main_loop ; b isn't set so just head back to the loop
pop af
dec a
cp -1
jr z,restore_hl
push af ; don't need this on the stack if we go to restore_hl, so place it after the jump
jr main_loop
restore_hl:
ld b,0 ; unset flag
pop hl
inc hl
; and continue executing into loop
main_loop:
ld a,(hl)
cp 0xff
jr z,wait
cp 0xfe
jr z,wait_n_times
cp 0xfd
jr z,end
cp 0xfc
jr z,jump
jr register_write
inc_loop:
inc hl
jr loop
wait:
call wait_for_raster_50hz_emu
jr inc_loop
wait_n_times: ; safe to assume that param isn't 0
push bc
inc hl
ld b,(hl)
wait_n_times_loop:
call wait_for_raster_50hz_emu
djnz wait_n_times_loop
pop bc
jr inc_loop
end:
jr end ; infinite loop
jump:
inc hl
ld d,(hl)
inc hl
ld e,(hl)
push hl
ld b,1 ; signal that we're calling a previous segment
ld a,N ; we want to execute N instructions before going back to where we left off
push af
ld hl,psg_begin
add hl,de
jr loop
register_write:
ld a,(hl)
out (c),a
; really we only need ld a,(hl) and out (0xa1),a, but let's poke around in the VRAM to make this program slightly less boring
; we'll modify the tile definitions of characters /, 0, ..., 9 (8 bytes each starting at 0x178) and just put in the same value we're writing to the PSG register
or a ; clear carry flag to make rla behave
; a = a*8 for vram write address
rla ; *2
rla ; *2 (*2*2 == *4)
rla ; *2 (*2*2*2 == *8)
ld d,a ; vram write address
inc hl
ld a,(hl)
ld e,a ; vram write value
out (0xa1),a
ld a,0x78
add a,d ; vram address low byte is 0x78 + (psg register)*8
; color change code currently commented out because it's not very pleasant to look at
; ; let's also change some colors when register 5 is written to, which doesn't appear to happen very often
; ; for register 5 a is 5*8 + 0x78 = 0xa0
; cp 0xa0
; jr nz,skip_color_change
; ld d,a
; ld a,e
; out (099h),a
; ld a,010000111b ; set register number (7)
; out (099h),a
; ld a,d
skip_color_change:
SetVdpWrite 1 a ; vram address high byte is 1 (full address: 0x178)
vpoke e
vpoke e
vpoke e
vpoke e
vpoke e
vpoke e
vpoke e
vpoke e
jr inc_loop
wait_for_raster:
in a,(0x99)
and 128
cp 128
jr nz,wait_for_raster
ret
wait_for_raster_50hz_emu:
; CPU clock is 3579545 Hz
; decrement and loop routine takes 36 instructions per loop run (wait_for_raster_50hz_emu_loop up to (not including) low_0)
; (https://www.overtakenbyevents.com/tstates/)
; want routine to finish in 1/50 or a second, so:
; 3579545/50/36=1988.636111111111, let's very scientifically, er, let's throw out that whole calculation and say 1650 because we have overhead and I have experimentally determined that to sound close enough to the original :p
; our overhead varies depending on code path. some rhythm problems are audible, but not _too_ terrible
ld de,1650
wait_for_raster_50hz_emu_loop:
dec de
ld a,e
cp 0
jr z,low_0
jr wait_for_raster_50hz_emu_loop
low_0:
ld a,d
cp 0
jr z,high_low_0
jr wait_for_raster_50hz_emu_loop
high_low_0:
ret
psg_begin:
include "foo.psg"
ds 010000h-$
Compiles with z80asm. Other assemblers might need some tweaks.
Bonus: converting PSG files to VGM
This is implemented in straight-forward C. Compilation: cc -o psg2vgm psg2vgm.c Execution: ./psg2vgm Dreamer\ -\ POPSA-2\ \(1994\).psg | xxd -r -p > foo.vgm
