Abdelmalek Abdesselam's suggestion that you try Groebner bases is reasonable for the degre-6 examples that mention in your update to the original question.
Let me spell out the steps explicitly to check whether two forms homogeneous f and g of the same degree in [x,y] are equivalent:
Let $a,b,c,d$ be the matrix-entry coordinates on ${\rm GL}_2$ and consider $f(ax+by,cx+dy)$.
Calculate $f(ax+by,cx+dy)-g(x,y)$ and extract the list $J$ of coefficients of $x^i y^j$. This list represents an ideal in $k[a,b,c,d,t]/(t(ad-bc)-1)$, the coordinate ring of ${\rm GL}_2$. The ideal corresponds to the subscheme of ${\rm GL}_2$ transporting $f$ into $g$, i.e. it gives the conditions on a section of ${\rm GL}_2$ for that section to transform $f$ into $g$.
Compute a Groebner basis of $K = J+[t(ad-bc)-1]$ in $k[a,b,c,d,t]$. If $K$ is not equivalent to $[1]$---if it is not the unit ideal---then there is a section of the transporter over $\overline{k}$ (Nullstellensatz). Computing such a section tells one how to transform $f$ into $g$. If $K$ is equivalent to $[1]$---if it is the unit ideal---then ${\rm GL}_2$ does not transform $f$ into $g$ over $\overline{k}$.
If you would prefer to work with a subgroup scheme $G$ of ${\rm GL}_2$, then add the generators of the ideal defining $G$ to $K$ before computing the Groebner basis.
I tried a pair of your forms, checking that $h(x,y)$ is equivalent to $f(x,y)$. Maple returned the following Groebner basis (lexicographic order t>a>b>c>d) instantly:
$$
[d^{12}+2118775924690448809984*d^6+93687714211574708957308664016389973143977984, 5*d^7+14472313805968991556993024*c+96131361823291542077440*d, d^7+20557263928933226643456*b+19226272364658308415488*d, -5*d+704*a, 1039*d^{10}-13614472272996909891715072*d^4+1144692921788849487441900451832894830569062400*t]
$$
This Groebner basis tells us to do the following to find the invertible matrix transforming h into f:
a. Solve the following degree-12 equation for $d$:
$$
d^{12}+2118775924690448809984d^6+93687714211574708957308664016389973143977984 = 0.
$$
Any root is fine.
b. Use the following equation to find $c$ from $d$:
$$
5d^7+96131361823291542077440d +14472313805968991556993024c= 0.
$$
c. Use the following equation to find $b$ from $d$:
$$
d^7+19226272364658308415488d +20557263928933226643456b= 0.
$$
d. Use the following equation to find $a$ from $d$:
$$
-5d+704a=0.
$$
The transporter scheme is defined over $k$ (which is $\mathbf{Q}$ in this case), and you can use it to study the fields over which $f$ and $h$ are equivalent, if you wish.
I tried the same procedure for transforming $g(x,y)$ into $f(x,y)$ and found that one could construct a matrix for the transformation as follows:
a. Solve the equation
$$
d^6-121740744925904896 = 0
$$
to find $d$.
b. Set $c=0$ and $b=0$.
c. Use the equation
$$
-5d+704a = 0
$$
to get $a$.
There is another type of transformation matrix with $a=0$ and $d=0$, whose description I omit. (The parameterizing scheme also has degree 6.)
In both cases, we get a degree-12 scheme for the total transporter, since your forms have automorphism group schemes of degree 12. (The transporter, when nonempty, is a bitorsor under the automorphism group schemes of source and target forms.)
This straightforward procedure could become unusable if you had a large number of forms to check or if you wanted to work with forms of higher degree. For studying a few forms of low degree, however, it is both speedy and easy---and it does not require looking up any invariants.
In any case, my question is if an algorithm can be designed without computing invariants.
– Jul 04 '14 at 13:42I have seen Olver's singular curve long time ago, but don't remember much of it. Do you know if anybody has tried to implement it somewhere? There are some methods or ideas to compare if two forms are equivalent under affine transformations, or even possibly under $GL_2 (k)$ using the minimal heights for forms. We'll see if anything good comes out of them!
– Jul 07 '14 at 20:27