(1)由\( \iint_{D} f(x,y)\text{d}x\text{d}y = 1 \)知,\( \int_{-1}^{1}\text{d}x\int_{x^{2}}^{1}ax^{2}y\text{d}y = 1 \),解得\( a = \frac{21}{4} \)。
(2)法一 令\( \begin{cases} z = x^{2}y, \\ v = y, \end{cases} \)由\( z = x^{2}y \)可知\( z,y \)同号,所以\( z,v \)同号,则\( \begin{cases} x = \sqrt{z/v}, \\ y = v \end{cases} \)或\( \begin{cases} x = -\sqrt{z/v}, \\ y = v. \end{cases} \)若\( \begin{cases} x = \sqrt{z/v}, \\ y = v, \end{cases} \) 则雅可比行列式为 \[ J_{1} = \begin{vmatrix} \frac{1}{2}(zv)^{-\frac{1}{2}} & -\frac{1}{2}z^{\frac{1}{2}}v^{-\frac{3}{2}} \\ 0 & 1 \end{vmatrix} = \frac{1}{2}(zv)^{-\frac{1}{2}}. \] 同理,若\( \begin{cases} x = -\sqrt{z/v}, \\ y = v, \end{cases} \) 则雅可比行列式为\( J_{2} = -\frac{1}{2}(zv)^{-\frac{1}{2}} \),所以\( (z,v) \)的概率密度为 \[ f(z,v) = \begin{cases} \frac{21}{4}z \cdot \frac{1}{2}(zv)^{-\frac{1}{2}} + \frac{21}{4}z \cdot \frac{1}{2}(zv)^{-\frac{1}{2}}, & 0 \leq z \leq v^{2} \leq 1, v > 0, \\ 0, & \text{其他} \end{cases} = \begin{cases} \frac{21}{4}\sqrt{z/v}, & 0 \leq z \leq v^{2} \leq 1, v > 0, \\ 0, & \text{其他}. \end{cases} \] 故\( Z \)的概率密度为 \[ f_{Z}(z) = \begin{cases} \int_{\sqrt{z}}^{1}\frac{21}{4}\sqrt{z/v}\text{d}v, & 0 \leq z \leq 1, \\ 0, & \text{其他} \end{cases} = \begin{cases} \frac{21}{2}z^{\frac{1}{2}} - \frac{21}{2}z^{\frac{3}{4}}, & 0 \leq z \leq 1, \\ 0, & \text{其他}. \end{cases} \] 法二 记\( Z \)的分布函数为\( F(z) \)。由题设知, 当\( z < 0 \)时,\( F(z) = 0 \);当\( z > 1 \)时,\( F(z) = 1 \); 当\( 0 \leq z \leq 1 \)时, \[ \begin{align*} F(z) &= P\{X^{2}Y \leq z\} = 1 - 2\left(\int_{z^{\frac{1}{4}}}^{1}\text{d}x\int_{x^{2}}^{\frac{z}{x^{2}}}\frac{21}{4}x^{2}y\text{d}y + \int_{z^{\frac{1}{4}}}^{1}\text{d}x\int_{x^{2}}^{1}\frac{21}{4}x^{2}y\text{d}y\right) \\ &= 1 - 2\left[\int_{z^{\frac{1}{4}}}^{1}\frac{21}{8}x^{2}\left(1 - \frac{z^{2}}{x^{4}}\right)\text{d}x + \int_{z^{\frac{1}{4}}}^{1}\frac{21}{8}x^{2}\left(1 - x^{4}\right)\text{d}x\right] \\ &= 1 - 2\left[\frac{21}{8}\left(\frac{x^{3}}{3} + \frac{z^{2}}{x}\right)\bigg|_{z^{\frac{1}{4}}}^{1} + \frac{21}{8}\left(\frac{1}{3}x^{3} - \frac{1}{7}x^{7}\right)\bigg|_{z^{\frac{1}{4}}}^{1}\right] \\ &= 1 - 2\left(\frac{1}{2} - \frac{7}{2}z^{\frac{3}{2}} + 3z^{\frac{7}{4}}\right) = 7z^{\frac{3}{2}} - 6z^{\frac{7}{4}}. \end{align*} \] 所以\( Z \)的概率密度为 \[ f_{Z}(z) = F'(z) = \begin{cases} \frac{21}{2}z^{\frac{1}{2}} - \frac{21}{2}z^{\frac{3}{4}}, & 0 \leq z \leq 1, \\ 0, & \text{其他}. \end{cases} \]
(3)当\( -1 < x < 1 \)时,\( f_{X}(x) = \int_{x^{2}}^{1}\frac{21}{4}x^{2}y\text{d}y = \frac{21}{8}x^{2}(1 - x^{4}) \),所以 \[ f_{Y|X}(y \mid x) = \frac{f(x,y)}{f_{X}(x)} = \begin{cases} \frac{2y}{1 - x^{4}}, & x^{2} < y < 1, \\ 0, & \text{其他}, \end{cases} \] 所以\( f_{Y|X}\left(y \mid x = \frac{1}{2}\right) = \begin{cases} \frac{32y}{15}, & \frac{1}{4} < y < 1, \\ 0, & \text{其他}. \end{cases} \)故\( E\left(Y \mid X = \frac{1}{2}\right) = \int_{\frac{1}{4}}^{1}\frac{32y^{2}}{15}\text{d}y = \frac{7}{10} \)。