Reconstructing sdb.txt

[Kaivan]

After spending quite some time looking through the obfuscated scripts, I've started questioning whether we can actually replace the obfuscated variables with something readable.

As some of you know, many times when reading through the scripts, we've been replacing variables with a variable name that describes the use of that particular variable as best as possible. This is done by replacing the Q*** variable with variablename_Q***.

While the above method works somewhat decently, I think that it is limited in terms of really getting a good idea of the variable's purpose. In that stead, I propose that we actually attempt to reconstruct the sdb.txt file as best as possible, using variable names that approximate the real variable name as closely as possible.

I believe that this is something that can be done, given enough time and some ingenuity with variable names. This stems from three different factors.

1. sdb.txt is in alphabetical order.

The alphabetical listing is the basic idea for this task. Without an alphabetical listing, it would be impossible to reconstruct the real (or very closely approximated) variable names. Sdb.txt uses what looks to be the ascii ordering for alphabetical listings.

2. alphabetical order + context of the script = possible variable names.

Having started this myself on the side, I have found that the context of a script does a lot to identify a possible name for a variable. Here's an example....

From the pet script, we have a function that is called when a player is added to the friend list of a given pet. This is the script in it's raw form:

Code: Select all

void Q458(object pet, object Q5CJ)
{
  list myBoss;
  if(!hasObjListVar(pet, "myBoss"))
  {
    setObjVar(pet, "myBoss", myBoss);
  }
  getObjListVar(myBoss, pet, "myBoss");
  if(!isInList(myBoss, Q5CJ))
  {
    appendToList(myBoss, Q5CJ);
  }
  setObjVar(pet, "myBoss", myBoss);
  return();
}

If we take a look at sdb.txt, the variable Q458 shows up here in the file:

Code: Select all

add
Q457
Q458
addBounty

Looking at the above function, the word boss appears all over the place. That gives us the context of dealing with bosses. Since we also know that this script has to do with adding an object to the boss list of a pet, we can guess that the original variable might have been addBoss. Notice that this variable actually fits in the alphabetical listing in sdb.txt, and it makes sense to be there.

Additionally, look at the variable Q5CJ. It appears here in sdb.txt:

Code: Select all

newType
Q5CC
Q5CD
Q5CE
Q5CF
Q5CG
Q5CH
Q5CI
Q5CJ
newcharges

Using the same concept as the addBoss variable, we can say that the variable name that Q5CJ represents is more than likely newboss. This too fits alphabetically with the sdb listing, and we now produce a resulting script that looks like this:

Code: Select all

void addBoss(object pet, object newboss)
{
  list myBoss;
  if(!hasObjListVar(pet, "myBoss"))
  {
    setObjVar(pet, "myBoss", myBoss);
  }
  getObjListVar(myBoss, pet, "myBoss");
  if(!isInList(myBoss, newboss))
  {
    appendToList(myBoss, newboss);
  }
  setObjVar(pet, "myBoss", myBoss);
  return();
}

3. Over time, as variables are discovered, sdb.txt will become easier to decipher.

This is a natural bonus to the project itself. As the listing becomes more filled, later variables will become easier to fill in. Of course, this does assume that the variable we have in a specific position is correct, or approximately correct, and we may find that at times we did have an incorrect variable when another variable that is much more obvious would throw that variable into an incorrect location. However, the overall trend will be towards an easier to decipher sdb file over time.

In any case, this is the idea as it stands. Thoughts?

[Batlin]

I've got a SDB Editor, which is quite old already but never released publicly.

It allows for quickly renaming the entries. Then using the Mass M Decompiler to recompile the scripts so they are updated.
Only thing to watch out for is
- avoiding names that collide with event names
- and function names (see the Command List http://uodemo.joinuo.com/?title=Command_List).
- avoid collisions with existing variable names (there is no quick way to tell which scripts uses a certain variable)

In my pov it's all about time, but with a few people working on it actively it can be done rather quick. Honestly, I wish we started doing this already a year ago .

I've attached the SDB Editor to this post. I suggest using SVN to update the SDB, so changes by multiple people can be merged with easy.

1. sdb.txt is in alphabetical order

I never noticed that, but definitly good to know.

[Batlin]

For the record, I wish to add that like Kaivan suggested that the string are sorted, but it's important to note that the strings are sorted case-sensitive. This means: 0 < 9 < A < Z < a < z.

[Batlin]

I've written a python script which will rename all Q-variables by finding the longest common substring between the last non-Q and first non-Q lines:

Code: Select all

import io
import sys

# open the file and start
f = io.open("sdb.txt", "r+")
newList = []
qlist = []
lastLine = ""
waserror = 0
for currentLine in f.readlines():
  # remove the enter
  if currentLine[-1] == '\n':
    currentLine = currentLine[0:-1]
  # start ignoring everything after an error occured
  if waserror != 0:
    qlist.append(currentLine)
    continue
  # ignore quoted strings
  if currentLine[0] == '"': # or currentLine[0:1] == 'L"':
    newList.append(currentLine)
    continue
  # is Q-string?
  isQ = currentLine[0] == 'Q' and len(currentLine) == 4
  # detect errors in the order/naming
  if not isQ and lastLine >= currentLine:
    # qlist is now used as error list :)
    qlist.append(currentLine)
    print("ERROR: " + currentLine + " >= " + lastLine, file=sys.stderr)
    waserror = 1
    continue
  if not isQ:
    # Find the longest common string starting from the first character
    a = lastLine
    b = currentLine
    lena = len(a)
    lenb = len(b)
    lenmin = min(lena, lenb)
    sharedlen = lenmin
    for i in range(1,lenmin+1):
      if a[0:i] != b[0:i]:
        sharedlen = i - 1
        break
    if sharedlen == 0:
      if a == "":
        sharedstring = "_" + b[0]
      else:
        sharedstring = a[0] + b[0]
    else:
      sharedstring = a[0:sharedlen]
    # add each renamed Q to the final list
    for q in qlist:
      newList.append(sharedstring + "__" + q)
    qlist = []
    # add the current line as is to the final list
    newList.append(currentLine)     
    lastLine = currentLine
  else:
    # add to the Qlist as-is
    qlist.append(currentLine)

for q in qlist:
  newList.append(q)

qlist = []

for newLine in newList:
  print(newLine)



NOTE: this one requires Python 3

[Dies Irae]

Could you post periodically in the svn the reconstructed file?

^_^